From tim.peters at gmail.com Mon Nov 1 02:26:33 2004 From: tim.peters at gmail.com (Tim Peters) Date: Mon Nov 1 02:26:40 2004 Subject: [Python-Dev] Tests broken on Windows In-Reply-To: <20041031195218.GA7395@surfboy> References: <1f7befae0410311127b31675c@mail.gmail.com> <20041031195218.GA7395@surfboy> Message-ID: <1f7befae04103117261e4d7f97@mail.gmail.com> [Johannes Gijsbers] > Hum, I didn't realize os.chmod() worked this differently on Windows. Well, it can't work the same. At the chmod() level, there are no "user", "group" or "world"/"other" concepts on Windows. There also aren't distinct "execute"/"search", "write" and "read" bits. There are "hidden", "system", and "archive" bits on Windows, which have no equivalents on Unix, and there's a "read only" bit on Windows, setting which is kind of like turning off 0222 on Unix. The only thing chmod() is useful for on Windows is fiddling the state of the "read only" bit. > Seems like I bumped into bug #755617 (os module: Need a better description > of "mode"). Per that bug, I googled for '_chmod msdn' and came up with the > attached patch. However, I don't have a Windows machine handy to test. Could > someone with a Windows machine try the patch? No cigar. The patched test appears to assume that if you strip write permission from a directory, that also makes it impossible to delete files within the directory. But directories on Windows aren't files -- they're directories <wink>. The attached patch allows the test to pass on WinXP, although I'm not sure about Win95/98/ME. If there are no objections, I'll check it in. I don't remember enough about Unix's equally obscure permission gimmicks to be sure that this patch will leave the test working on non-Windows boxes. -------------- next part -------------- Index: Lib/test/test_shutil.py =================================================================== RCS file: /cvsroot/python/python/dist/src/Lib/test/test_shutil.py,v retrieving revision 1.8 diff -u -r1.8 test_shutil.py --- Lib/test/test_shutil.py 31 Oct 2004 12:05:31 -0000 1.8 +++ Lib/test/test_shutil.py 1 Nov 2004 01:10:32 -0000 @@ -1,4 +1,3 @@ - # Copyright (C) 2003 Python Software Foundation import unittest @@ -20,21 +19,28 @@ def test_on_error(self): self.errorState = 0 os.mkdir(TESTFN) - f = open(os.path.join(TESTFN, 'a'), 'w') + self.childpath = os.path.join(TESTFN, 'a') + f = open(self.childpath, 'w') f.close() - # Make TESTFN unwritable. - os.chmod(TESTFN, stat.S_IRUSR) + old_dir_mode = os.stat(TESTFN).st_mode + old_child_mode = os.stat(self.childpath).st_mode + # Make unwritable. + os.chmod(self.childpath, stat.S_IREAD) + os.chmod(TESTFN, stat.S_IREAD) shutil.rmtree(TESTFN, onerror=self.check_args_to_onerror) - # Make TESTFN writable again. - os.chmod(TESTFN, stat.S_IRWXU) + # Make writable again. + os.chmod(TESTFN, old_dir_mode) + os.chmod(self.childpath, old_child_mode) + + # Clean up. shutil.rmtree(TESTFN) def check_args_to_onerror(self, func, arg, exc): if self.errorState == 0: self.assertEqual(func, os.remove) - self.assertEqual(arg, os.path.join(TESTFN, 'a')) + self.assertEqual(arg, self.childpath) self.assertEqual(exc[0], OSError) self.errorState = 1 else: From jlg at dds.nl Mon Nov 1 03:38:47 2004 From: jlg at dds.nl (Johannes Gijsbers) Date: Mon Nov 1 03:36:47 2004 Subject: [Python-Dev] Tests broken on Windows In-Reply-To: <1f7befae04103117261e4d7f97@mail.gmail.com> References: <1f7befae0410311127b31675c@mail.gmail.com> <20041031195218.GA7395@surfboy> <1f7befae04103117261e4d7f97@mail.gmail.com> Message-ID: <20041101023847.GA17801@surfboy> On Sun, Oct 31, 2004 at 08:26:33PM -0500, Tim Peters wrote: > The attached patch allows the test to pass on WinXP, although I'm not > sure about Win95/98/ME. If there are no objections, I'll check it in. > I don't remember enough about Unix's equally obscure permission > gimmicks to be sure that this patch will leave the test working on > non-Windows boxes. It works on my Debian machine, so no objections from me. Cheers, Johannes From python at rcn.com Mon Nov 1 04:02:52 2004 From: python at rcn.com (Raymond Hettinger) Date: Mon Nov 1 04:05:01 2004 Subject: [Python-Dev] Tests broken on Windows In-Reply-To: <20041101023847.GA17801@surfboy> Message-ID: <006601c4bfbf$478258c0$e841fea9@oemcomputer> > On Sun, Oct 31, 2004 at 08:26:33PM -0500, Tim Peters wrote: > > The attached patch allows the test to pass on WinXP, although I'm not > > sure about Win95/98/ME. If there are no objections, I'll check it in. > > I don't remember enough about Unix's equally obscure permission > > gimmicks to be sure that this patch will leave the test working on > > non-Windows boxes. > > It works on my Debian machine, so no objections from me. Run fine on WinME. Am still getting a failure for test_traceback.py's test_bug737473. Raymond From tim.peters at gmail.com Mon Nov 1 06:52:03 2004 From: tim.peters at gmail.com (Tim Peters) Date: Mon Nov 1 06:52:04 2004 Subject: [Python-Dev] Tests broken on Windows In-Reply-To: <006601c4bfbf$478258c0$e841fea9@oemcomputer> References: <20041101023847.GA17801@surfboy> <006601c4bfbf$478258c0$e841fea9@oemcomputer> Message-ID: <1f7befae04103121526ff7a94c@mail.gmail.com> [Raymond Hettinger] > Run fine on WinME. > > Am still getting a failure for test_traceback.py's test_bug737473. Fixed that already for WinXP. No current failures in the -uall test suite. Is there a bug report for this ME problem? Exactly how does it fail? Does it work if you force the test to take the time.sleep() path instead of the os.utime() path? Is posixmodule.c's posix_utime() taking the utime() or _wutime() path on ME? If the latter, does it work if you force it to take the utime() path instead? MS docs claim utime() and _wutime() work "under Windows 98/Me and Windows NT/2000/XP", but they could be lying: <http://msdn.microsoft.com/library/default.asp?url=/library/en-us/vclib/html/_crt__utime.2c_._wutime.asp> That link has a wonderful C example: if you study the sample output, it shows that the sample code they ran had no visible effect <wink>. From python at rcn.com Mon Nov 1 07:50:29 2004 From: python at rcn.com (Raymond Hettinger) Date: Mon Nov 1 07:52:40 2004 Subject: [Python-Dev] Tests broken on Windows In-Reply-To: <1f7befae04103121526ff7a94c@mail.gmail.com> Message-ID: <007c01c4bfdf$138f7f00$e841fea9@oemcomputer> > Fixed that already for WinXP. No current failures in the -uall test > suite. Is there a bug report for this ME problem? Exactly how does > it fail? Traceback (most recent call last): File "test_traceback.py", line 79, in test_bug737473 test_bug737473.test() File "c:\windows\temp\tmpsj7w5a\test_bug737473.py", line 2, in test ValueError > Does it work if you force the test to take the time.sleep() > path instead of the os.utime() path? Yes. > Is posixmodule.c's posix_utime() > taking the utime() or _wutime() path on ME? If the latter, does it > work if you force it to take the utime() path instead? MS docs claim > utime() and _wutime() work "under Windows 98/Me and Windows > NT/2000/XP", but they could be lying: > > <http://msdn.microsoft.com/library/default.asp?url=/library/en- > us/vclib/html/_crt__utime.2c_._wutime.asp> > > That link has a wonderful C example: if you study the sample output, > it shows that the sample code they ran had no visible effect <wink>. Am challenged for time in the next couple of days. Will delve deeper at the first opportunity. For the time being, I think it would be best to use sleep() instead of utime(). While it costs three seconds, at least we know it to be a platform independent test. It is rather late in the game to be debugging OS specific problems introduced by a new check-in. Raymond From hyeshik at gmail.com Mon Nov 1 09:19:24 2004 From: hyeshik at gmail.com (Hye-Shik Chang) Date: Mon Nov 1 09:19:31 2004 Subject: [Python-Dev] Tests broken on Windows In-Reply-To: <007c01c4bfdf$138f7f00$e841fea9@oemcomputer> References: <1f7befae04103121526ff7a94c@mail.gmail.com> <007c01c4bfdf$138f7f00$e841fea9@oemcomputer> Message-ID: <4f0b69dc04110100194b7335f4@mail.gmail.com> On Mon, 1 Nov 2004 01:50:29 -0500, Raymond Hettinger <python@rcn.com> wrote: > For the time being, I think it would be best to use sleep() instead of > utime(). While it costs three seconds, at least we know it to be a > platform independent test. It is rather late in the game to be > debugging OS specific problems introduced by a new check-in. Okay. I am convinced about using sleep() always. And two seconds will be enough to wait for new timestamp value. :) Will check it in right now. Hye-Shik From walter at livinglogic.de Mon Nov 1 15:17:22 2004 From: walter at livinglogic.de (=?ISO-8859-15?Q?Walter_D=F6rwald?=) Date: Mon Nov 1 15:17:25 2004 Subject: [Python-Dev] Code coverage tool updated Message-ID: <41864572.4000806@livinglogic.de> The Python code coverage tool available at http://coverage.livinglogic.de/ has been updated. (It runs the Python test suite once a month and makes the resulting code coverage info available as a web application). The backend is now Oracle instead of Postgres and is on a much faster machine. The frontend has been replaced too, so the result should be much faster and more flexible. Bye, Walter D?rwald From tim.peters at gmail.com Mon Nov 1 15:40:55 2004 From: tim.peters at gmail.com (Tim Peters) Date: Mon Nov 1 15:40:58 2004 Subject: [Python-Dev] Tests broken on Windows In-Reply-To: <007c01c4bfdf$138f7f00$e841fea9@oemcomputer> References: <1f7befae04103121526ff7a94c@mail.gmail.com> <007c01c4bfdf$138f7f00$e841fea9@oemcomputer> Message-ID: <1f7befae0411010640413d9148@mail.gmail.com> > Exactly how does it fail? [Raymond Hettinger, on bug737473] > Traceback (most recent call last): > File "test_traceback.py", line 79, in test_bug737473 > test_bug737473.test() > File "c:\windows\temp\tmpsj7w5a\test_bug737473.py", line 2, in test > ValueError That helps. >> Does it work if you force the test to take the time.sleep() >> path instead of the os.utime() path? > Yes. Progress <wink>. ... > Am challenged for time in the next couple of days. Will delve deeper at > the first opportunity. > > For the time being, I think it would be best to use sleep() instead of > utime(). While it costs three seconds, at least we know it to be a > platform independent test. It is rather late in the game to be > debugging OS specific problems introduced by a new check-in. The checkin isn't that new anymore. Just try changing past = time.time() - 3 to past = time.time() - 6 I added a note to <http://www.python.org/1057992> explaining why that may matter on FAT or FAT32 systems. From jlg at dds.nl Mon Nov 1 15:40:27 2004 From: jlg at dds.nl (Johannes Gijsbers) Date: Mon Nov 1 16:09:29 2004 Subject: [Python-Dev] Bug Day 4: this Sunday Message-ID: <20041101144027.GA16108@surfboy> The fourth Python Bug Day is coming up this Sunday, November 7 2004. When? ===== This Sunday, November 7, 2004. I'll be online at least from 11AM to 6PM CET (10AM to 5PM GMT/ 6AM to 1PM EDT). That's a rather short time for the people in the USian timezones, so I would like to see someone with CVS access from those zones extend the bug day. Anyone interested? Where? ====== * The #python-dev channel on irc.freenode.net. * In real life, at the Amaze office: Korte Leidsedwarsstraat 12 1017 RC Amsterdam tel: 020-4688336 More information ================ For instructions and more information, see http://www.python.org/moin/PythonBugDay Cheers, Johannes From mwh at python.net Mon Nov 1 18:22:03 2004 From: mwh at python.net (Michael Hudson) Date: Mon Nov 1 18:22:06 2004 Subject: [Python-Dev] Code coverage tool updated In-Reply-To: <41864572.4000806@livinglogic.de> ( =?iso-8859-1?q?Walter_D=F6rwald's_message_of?= "Mon, 01 Nov 2004 15:17:22 +0100") References: <41864572.4000806@livinglogic.de> Message-ID: <2mu0s95vt0.fsf@starship.python.net> Walter D?rwald <walter@livinglogic.de> writes: > The Python code coverage tool available at > http://coverage.livinglogic.de/ has been updated. (It runs the > Python test suite once a month and makes the resulting code > coverage info available as a web application). > > The backend is now Oracle instead of Postgres and is on a much > faster machine. The frontend has been replaced too, so the result > should be much faster and more flexible. Using the back button in a browser seems to break it fairly badly -- if you click on a file that has some results, then click 'back' you get a java.lang.NullPointerException! If you click on a file _without_ results and click 'back' you merely get a message in German that I don't understand... Cheers, mwh -- Programming languages should be designed not by piling feature on top of feature, but by removing the weaknesses and restrictions that make the additional features appear necessary. -- Revised(5) Report on the Algorithmic Language Scheme From skip at pobox.com Mon Nov 1 23:01:56 2004 From: skip at pobox.com (Skip Montanaro) Date: Mon Nov 1 23:02:20 2004 Subject: [Python-Dev] Code coverage tool updated In-Reply-To: <41864572.4000806@livinglogic.de> References: <41864572.4000806@livinglogic.de> Message-ID: <16774.45652.835103.327656@montanaro.dyndns.org> Walter> The Python code coverage tool available at Walter> http://coverage.livinglogic.de/ has been updated. (It runs the Walter> Python test suite once a month and makes the resulting code Walter> coverage info available as a web application). This looks very interesting, but the results seem a bit mystifying. For example, it says that Lib/Queue.py has 169 lines, none of which are coverable. It also shows that no lines were covered, which seems odd considering the fact the distribution does have a test_queue.py test module. Skip From satyarth.negi at gmail.com Tue Nov 2 10:13:12 2004 From: satyarth.negi at gmail.com (Satyarth Negi) Date: Tue Nov 2 10:13:16 2004 Subject: [Python-Dev] Help Needed ..Urgent Plzz Message-ID: <546a6a000411020113195cb03f@mail.gmail.com> Hello , I have a problem with a peice of code : ......... self.child = pexpect.spawn('ssh -l ' + user +' ' + host) self.child.setmaxread (1000) self.child.expect('[pP]assword:', 30) self.child.sendline(password) ........ When i run the script it gives me following error : " Exception exceptions.OSError: (10, 'No child processes') in <bound method spawn.__del__ of <pexpect.spawn instance at 0x403d938c>> ignored" Then I added , self.child.close(False) before 3rd line , but even that is not working . Please help , im totaly stuck . Thanx in anticipation . Regards Satyarth Negi India From walter at livinglogic.de Tue Nov 2 12:58:21 2004 From: walter at livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=) Date: Tue Nov 2 12:58:25 2004 Subject: [Python-Dev] Code coverage tool updated In-Reply-To: <16774.45652.835103.327656@montanaro.dyndns.org> References: <41864572.4000806@livinglogic.de> <16774.45652.835103.327656@montanaro.dyndns.org> Message-ID: <4187765D.3000705@livinglogic.de> Skip Montanaro wrote: > Walter> The Python code coverage tool available at > Walter> http://coverage.livinglogic.de/ has been updated. (It runs the > Walter> Python test suite once a month and makes the resulting code > Walter> coverage info available as a web application). > > This looks very interesting, but the results seem a bit mystifying. For > example, it says that Lib/Queue.py has 169 lines, none of which are > coverable. It also shows that no lines were covered, which seems odd > considering the fact the distribution does have a test_queue.py test module. The same happened to string.py. The run from yesterday *does* include string.py as a covered file, but only two lines were covered, which is even stranger. Maybe the problem is the fact, that I'm using an old version of trace.py? I'll updated trace.py and rerun the job. Bye, Walter D?rwald From walter at livinglogic.de Tue Nov 2 12:58:35 2004 From: walter at livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=) Date: Tue Nov 2 12:58:38 2004 Subject: [Python-Dev] Code coverage tool updated In-Reply-To: <2mu0s95vt0.fsf@starship.python.net> References: <41864572.4000806@livinglogic.de> <2mu0s95vt0.fsf@starship.python.net> Message-ID: <4187766B.7060407@livinglogic.de> Michael Hudson wrote: > Walter D?rwald <walter@livinglogic.de> writes: > > >>The Python code coverage tool available at >>http://coverage.livinglogic.de/ has been updated. (It runs the >>Python test suite once a month and makes the resulting code >>coverage info available as a web application). >> >>The backend is now Oracle instead of Postgres and is on a much >>faster machine. The frontend has been replaced too, so the result >>should be much faster and more flexible. > > Using the back button in a browser seems to break it fairly badly -- > if you click on a file that has some results, then click 'back' you > get a java.lang.NullPointerException! I can reproduce that. I've forwarded this to the developer (who is currently on vacation and will be back in December). > If you click on a file > _without_ results and click 'back' you merely get a message in German > that I don't understand... This message basically says "Don't click the back button!". Bye, Walter D?rwald From FBatista at uniFON.com.ar Tue Nov 2 13:35:49 2004 From: FBatista at uniFON.com.ar (Batista, Facundo) Date: Tue Nov 2 13:37:26 2004 Subject: [Python-Dev] Help Needed ..Urgent Plzz Message-ID: <A128D751272CD411BC9200508BC2194D053C7B40@escpl.tcp.com.ar> [Satyarth Negi] #- Hello , I have a problem with a peice of code : #- #- ......... #- ........ Satyarth, this list is for development of Python itself, not for development of programs in Python. This post should go to the standard python newsgroup, comp.lang.python, which can be accessed as a news or as a mailing list (for further reference and ways to subscribe, check http://www.python.org/community/lists.html). . Facundo -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20041102/3c69cf07/attachment.html From anthony at interlink.com.au Tue Nov 2 13:46:38 2004 From: anthony at interlink.com.au (Anthony Baxter) Date: Tue Nov 2 13:46:16 2004 Subject: [Python-Dev] TRUNK FROZEN for 2.4b2 release from 0000UTC 3rd November (tomorrrow) Message-ID: <418781AE.70303@interlink.com.au> The trunk should be considered frozen for the 2.4b2 release from 0000 UTC tomorrow, Wed 3rd November. That's (if my sums are correct) 2nd November from 1900 US Eastern time, and 1600 US Pacific time. Or, to put it another way, in about 11 hours from when I'm sending this message. As usual, if you're not in the set (Martin, Fred, Anthony) please hold off on any checkins until the b2 release is done. Thanks, Anthony -- Anthony Baxter <anthony@interlink.com.au> It's never too late to have a happy childhood. From anthony at interlink.com.au Tue Nov 2 13:48:49 2004 From: anthony at interlink.com.au (Anthony Baxter) Date: Tue Nov 2 13:49:22 2004 Subject: [Python-Dev] Bug Day 4: this Sunday In-Reply-To: <20041101144027.GA16108@surfboy> References: <20041101144027.GA16108@surfboy> Message-ID: <41878231.9060001@interlink.com.au> Johannes Gijsbers wrote: > This Sunday, November 7, 2004. I'll be online at least from 11AM to 6PM CET > (10AM to 5PM GMT/ 6AM to 1PM EDT). That's a rather short time for the people in > the USian timezones, so I would like to see someone with CVS access from those > zones extend the bug day. Anyone interested? Could I request that people are _really_ careful about checking in bugs that might lead to breakage, as this will be between 2.4b2, and what I _hope_ will be 2.4rc1. Thanks, Anthony -- Anthony Baxter <anthony@interlink.com.au> It's never too late to have a happy childhood. From walter at livinglogic.de Tue Nov 2 18:55:40 2004 From: walter at livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=) Date: Tue Nov 2 18:55:44 2004 Subject: [Python-Dev] Code coverage tool updated In-Reply-To: <4187765D.3000705@livinglogic.de> References: <41864572.4000806@livinglogic.de> <16774.45652.835103.327656@montanaro.dyndns.org> <4187765D.3000705@livinglogic.de> Message-ID: <4187CA1C.9050508@livinglogic.de> Walter D?rwald wrote: > Skip Montanaro wrote: > >> Walter> The Python code coverage tool available at >> Walter> http://coverage.livinglogic.de/ has been updated. (It runs >> the >> Walter> Python test suite once a month and makes the resulting code >> Walter> coverage info available as a web application). >> >> This looks very interesting, but the results seem a bit mystifying. For >> example, it says that Lib/Queue.py has 169 lines, none of which are >> coverable. It also shows that no lines were covered, which seems odd >> considering the fact the distribution does have a test_queue.py test >> module. > > > The same happened to string.py. The run from yesterday *does* include > string.py as a covered file, but only two lines were covered, which is > even stranger. > > Maybe the problem is the fact, that I'm using an old version of > trace.py? I'll updated trace.py and rerun the job. This doesn't help. The following assert in trace.py raises an AssertionError: assert filename.endswith('.py') I've added print statements to find_executable_linenos() and got the following: [...] /home/coverage/LivingLogic/Python/PythonCodeCoverage/jobs/python/dist/src/Lib/sre_compile.py /home/coverage/LivingLogic/Python/PythonCodeCoverage/jobs/python/dist/src/Lib/test/__init__.py <doctest _threading_local[7]> Traceback (most recent call last): File "../../../trace.py", line 790, in ? main() File "../../../trace.py", line 787, in main results.write_results(missing, summary=summary, coverdir=coverdir) File "../../../trace.py", line 301, in write_results lnotab = find_executable_linenos(filename) File "../../../trace.py", line 420, in find_executable_linenos assert filename.endswith('.py') AssertionError So what is <doctest _threading_local[7]>? Bye, Walter D?rwald From tim.peters at gmail.com Tue Nov 2 19:53:59 2004 From: tim.peters at gmail.com (Tim Peters) Date: Tue Nov 2 19:54:12 2004 Subject: [Python-Dev] Code coverage tool updated In-Reply-To: <4187CA1C.9050508@livinglogic.de> References: <41864572.4000806@livinglogic.de> <16774.45652.835103.327656@montanaro.dyndns.org> <4187765D.3000705@livinglogic.de> <4187CA1C.9050508@livinglogic.de> Message-ID: <1f7befae04110210537c35e514@mail.gmail.com> [Walter D?rwald] > This doesn't help. The following assert in trace.py raises > an AssertionError: > assert filename.endswith('.py') > > I've added print statements to find_executable_linenos() > and got the following: > > [...] > /home/coverage/LivingLogic/Python/PythonCodeCoverage/jobs/python/dist/src/Lib/sre_compile.py > /home/coverage/LivingLogic/Python/PythonCodeCoverage/jobs/python/dist/src/Lib/test/__init__.py > <doctest _threading_local[7]> > Traceback (most recent call last): > File "../../../trace.py", line 790, in ? > main() > File "../../../trace.py", line 787, in main > results.write_results(missing, summary=summary, coverdir=coverdir) > File "../../../trace.py", line 301, in write_results > lnotab = find_executable_linenos(filename) > File "../../../trace.py", line 420, in find_executable_linenos > assert filename.endswith('.py') > AssertionError > > So what is <doctest _threading_local[7]>? I haven't followed this thread, but can answer that question literally <wink>: it's code synthesized for the seventh doctest example in Lib/_threading_local.py's module docstring. The file name is constructed by this line in doctest.py: # Use a special filename for compile(), so we can retrieve # the source code during interactive debugging (see # __patched_linecache_getlines). filename = '<doctest %s[%d]>' % (test.name, examplenum) From bac at OCF.Berkeley.EDU Tue Nov 2 21:24:38 2004 From: bac at OCF.Berkeley.EDU (Brett C.) Date: Tue Nov 2 21:25:01 2004 Subject: [Python-Dev] TRUNK FROZEN for 2.4b2 release from 0000UTC 3rd November (tomorrrow) In-Reply-To: <418781AE.70303@interlink.com.au> References: <418781AE.70303@interlink.com.au> Message-ID: <4187ED06.70104@ocf.berkeley.edu> Anthony Baxter wrote: > The trunk should be considered frozen for the 2.4b2 release from 0000 > UTC tomorrow, Wed 3rd November. That's (if my sums are correct) 2nd > November from 1900 US Eastern time, and 1600 US Pacific time. Or, > to put it another way, in about 11 hours from when I'm sending this > message. > Is a deprecation of a function okay in the Mac C API? Specifically, PyMac_GetAppletScriptFile() is not being used by any known code and is a problem for http://www.python.org/sf/1035255 (removes dependency of compiling against some frameworks for the core on OS X). Since I didn't feel totally safe applying the patch for b2 since it changed compilation and I didn't get in early enough to get testing from CVS users I was planning on deprecating PyMac_GetAppletScriptFile() for 2.4 and removing it in 2.5 so I could apply the patch then. But it is a feature change so I wanted to get clearance first. As I said, to the best of my and Bob Ippolito's knowledge no one is using the function and it is a hinderance in removing unneeded linking against the core on OS X. Sorry I didn't do this sooner but I didn't think about deprecating the thing until late last night. -Brett From bob at redivi.com Tue Nov 2 21:46:19 2004 From: bob at redivi.com (Bob Ippolito) Date: Tue Nov 2 21:46:33 2004 Subject: [Python-Dev] TRUNK FROZEN for 2.4b2 release from 0000UTC 3rd November (tomorrrow) In-Reply-To: <4187ED06.70104@ocf.berkeley.edu> References: <418781AE.70303@interlink.com.au> <4187ED06.70104@ocf.berkeley.edu> Message-ID: <3FEE83B6-2D10-11D9-955F-000A95BA5446@redivi.com> On Nov 2, 2004, at 15:24, Brett C. wrote: > Anthony Baxter wrote: >> The trunk should be considered frozen for the 2.4b2 release from 0000 >> UTC tomorrow, Wed 3rd November. That's (if my sums are correct) 2nd >> November from 1900 US Eastern time, and 1600 US Pacific time. Or, >> to put it another way, in about 11 hours from when I'm sending this >> message. > > Is a deprecation of a function okay in the Mac C API? Specifically, > PyMac_GetAppletScriptFile() is not being used by any known code and is > a problem for http://www.python.org/sf/1035255 (removes dependency of > compiling against some frameworks for the core on OS X). Since I > didn't feel totally safe applying the patch for b2 since it changed > compilation and I didn't get in early enough to get testing from CVS > users I was planning on deprecating PyMac_GetAppletScriptFile() for > 2.4 and removing it in 2.5 so I could apply the patch then. > > But it is a feature change so I wanted to get clearance first. As I > said, to the best of my and Bob Ippolito's knowledge no one is using > the function and it is a hinderance in removing unneeded linking > against the core on OS X. > > Sorry I didn't do this sooner but I didn't think about deprecating the > thing until late last night. If it's absolutely necessary to keep this undocumented function that *nobody* uses in the API, I'll write a version that delegates the functionality to an extension module that has business linking to said frameworks. I really want to see this change for 2.4. -bob From anthony at interlink.com.au Wed Nov 3 03:34:39 2004 From: anthony at interlink.com.au (Anthony Baxter) Date: Wed Nov 3 03:34:14 2004 Subject: [Python-Dev] TRUNK FROZEN for 2.4b2 release from 0000UTC 3rd November (tomorrrow) In-Reply-To: <4187ED06.70104@ocf.berkeley.edu> References: <418781AE.70303@interlink.com.au> <4187ED06.70104@ocf.berkeley.edu> Message-ID: <418843BF.7020107@interlink.com.au> Brett C. wrote: > Anthony Baxter wrote: > >> The trunk should be considered frozen for the 2.4b2 release from 0000 >> UTC tomorrow, Wed 3rd November. That's (if my sums are correct) 2nd >> November from 1900 US Eastern time, and 1600 US Pacific time. Or, >> to put it another way, in about 11 hours from when I'm sending this >> message. >> > > Is a deprecation of a function okay in the Mac C API? Specifically, > PyMac_GetAppletScriptFile() is not being used by any known code and is a > problem for http://www.python.org/sf/1035255 (removes dependency of > compiling against some frameworks for the core on OS X). Since I didn't > feel totally safe applying the patch for b2 since it changed compilation > and I didn't get in early enough to get testing from CVS users I was > planning on deprecating PyMac_GetAppletScriptFile() for 2.4 and removing > it in 2.5 so I could apply the patch then. It sounds like it's fine to put in once the trunk is unfrozen. -- Anthony Baxter <anthony@interlink.com.au> It's never too late to have a happy childhood. From bac at OCF.Berkeley.EDU Wed Nov 3 04:43:24 2004 From: bac at OCF.Berkeley.EDU (Brett C.) Date: Wed Nov 3 04:43:56 2004 Subject: [Python-Dev] TRUNK FROZEN for 2.4b2 release from 0000UTC 3rd November (tomorrrow) In-Reply-To: <418843BF.7020107@interlink.com.au> References: <418781AE.70303@interlink.com.au> <4187ED06.70104@ocf.berkeley.edu> <418843BF.7020107@interlink.com.au> Message-ID: <418853DC.10501@ocf.berkeley.edu> Anthony Baxter wrote: > Brett C. wrote: > >> Anthony Baxter wrote: >> >>> The trunk should be considered frozen for the 2.4b2 release from 0000 >>> UTC tomorrow, Wed 3rd November. That's (if my sums are correct) 2nd >>> November from 1900 US Eastern time, and 1600 US Pacific time. Or, >>> to put it another way, in about 11 hours from when I'm sending this >>> message. >>> >> >> Is a deprecation of a function okay in the Mac C API? Specifically, >> PyMac_GetAppletScriptFile() is not being used by any known code and is >> a problem for http://www.python.org/sf/1035255 (removes dependency of >> compiling against some frameworks for the core on OS X). Since I >> didn't feel totally safe applying the patch for b2 since it changed >> compilation and I didn't get in early enough to get testing from CVS >> users I was planning on deprecating PyMac_GetAppletScriptFile() for >> 2.4 and removing it in 2.5 so I could apply the patch then. > > > It sounds like it's fine to put in once the trunk is unfrozen. > OK, great. I will add the deprecation warning once the trunk is unfrozen and plan on applying the full patch for removing the compilation once 2.5 is open for work. Sorry I couldn't get my act together fast enough to get this in for 2.4, Bob. -Brett From tim.peters at gmail.com Wed Nov 3 05:40:05 2004 From: tim.peters at gmail.com (Tim Peters) Date: Wed Nov 3 05:40:10 2004 Subject: [Python-Dev] weakref gc semantics Message-ID: <1f7befae04110220402077a6fa@mail.gmail.com> This example bothers me a little, under current Python CVS: """ import gc, weakref def C_gone(ignored): print "An object of type C went away." class C: def __init__(self): self.wr = weakref.ref(self, C_gone) print "creating non-cyclic C instance" c = C() print "creating cyclic C instance and destroying old one" c = C() # triggers the print c.loop = c print "getting rid of the cyclic one" c = None # doesn't trigger the print gc.collect() # ditto """ Output: creating non-cyclic C instance creating cyclic C instance and destroying old one An object of type C went away. getting rid of the cyclic one An instance c of C gets a strong reference (self.wr) to a weak reference W to itself. At the end, c is in a cycle, but W isn't. W ends up reachable only *from* stuff in a cycle (namely c). gc suppresses the callback then because W is unreachable, not really because W is in a cycle. If self.wr had a __del__ method instead, it would have been executed. So it bothers me a little that the callback isn't: a reachable callback can have visible effects, and there really isn't ambiguity about which of c and W "dies first" (c does) in the example. The question is whether gc should really be looking at whether the weakref *callback* is reachable, regardless of whether the weakref itself is reachable (if the weakref is reachable, its callback is too, and 2.4b2 invokes it -- the only case in question is the one in the example, where the weakref and the weakref's referent are unreachable but the weakref's callback is reachable). Pythons 2.2.3 and 2.3.4 produce the same output, so no way do I want to change Python 2.4 at this late stage. It's a question for 2.5. It gets more complicated than the above, of course. For example, we can end up with weakrefs *in* cycles, with reachable callbacks, and then there's no good way to decide which callback to invoke first. That's the same ordering problem we have when objects with __del__ methods are in trash cycles. It's not intractable, though, because gc invokes weakref callbacks before tp_clear has been invoked on anything, and reachable weakref callbacks can't resurrect any trash (anything they can access is, by definition of "reachable", not trash). A hidden agenda is that I like weakref callbacks a lot more than __del__ methods, and would like to see them strong enough so that Python 3 could leave __del__ out. Planting a weakref to self *in* self (as in the example at the top) is one way to approach critical-resource cleanup in a __del__-free world (a weakref subclass could remember the critical resource when an instance was constructed, and the callback could access the critical resource to clean it up -- but, as the example shows, that doesn't work now if the containing instance ends up in cyclic trash; but there are other ways that do work). From just at letterror.com Wed Nov 3 08:38:54 2004 From: just at letterror.com (Just van Rossum) Date: Wed Nov 3 08:39:39 2004 Subject: [Python-Dev] TRUNK FROZEN for 2.4b2 release from 0000UTC 3rd November (tomorrrow) In-Reply-To: <418853DC.10501@ocf.berkeley.edu> Message-ID: <r01050400-1026-6FEC84FE2D6B11D9ADC9003065D5E7E4@[10.0.0.23]> Brett C. wrote: > OK, great. I will add the deprecation warning once the trunk is > unfrozen and plan on applying the full patch for removing the > compilation once 2.5 is open for work. > > Sorry I couldn't get my act together fast enough to get this in for > 2.4, Bob. I don't get it. This is about an undocumented Mac-specific API function that noone uses (not even Python itself) which was introduced in 2.3. It just needs to be ripped out. I don't mind if it's done after b2, but I agree with Bob that it needs to be taken out before 2.4 final. Jack, would you agree? Just From anthony at interlink.com.au Wed Nov 3 08:49:34 2004 From: anthony at interlink.com.au (Anthony Baxter) Date: Wed Nov 3 08:50:14 2004 Subject: [Python-Dev] TRUNK FROZEN for 2.4b2 release from 0000UTC 3rd November (tomorrrow) In-Reply-To: <r01050400-1026-6FEC84FE2D6B11D9ADC9003065D5E7E4@[10.0.0.23]> References: <r01050400-1026-6FEC84FE2D6B11D9ADC9003065D5E7E4@[10.0.0.23]> Message-ID: <41888D8E.5050705@interlink.com.au> Just van Rossum wrote: > I don't get it. This is about an undocumented Mac-specific API function > that noone uses (not even Python itself) which was introduced in 2.3. It > just needs to be ripped out. I don't mind if it's done after b2, but I > agree with Bob that it needs to be taken out before 2.4 final. Jack, > would you agree? FWIW, I don't mind having this done either way... From anthony at interlink.com.au Wed Nov 3 12:16:49 2004 From: anthony at interlink.com.au (Anthony Baxter) Date: Wed Nov 3 13:19:52 2004 Subject: [Python-Dev] interlocking dependencies on the path to a release Message-ID: <4188BE21.6040205@interlink.com.au> So I've been documenting the current state of the release process, and it's a bit messier than I'd like - particularly with people scattered across timezones. Here's the list of things that need to happen, and what depends on each. 1. Include/patchlevel.h 2. Doc build (depends on 1.) 3. Misc/NEWS, Lib/idlelib/NEWS.txt, Lib/idlelib/idlever.py 4. PCbuild/BUILDno.txt, pythoncore.dsp 5. Tag the tree (depends on 1, 3, 5) 6. export/build the tarballs (depends on 5) 7. Build the windows release (depends on 2, 5) 8. Update the webpages, sign the files (depends on 6, 7) 9. Send the email (depends on 8) Stage 2 is a Fred thing, stages 4 and 7 are Martin, most of the rest are things I do, although Fred will often do 1 because he needs it to do 2, and the timezone thing makes it easier for him to just do it. (*) The names 'Fred', 'Martin', and 'I' are just the names of the current set of release people - feel free to substitute roles in there instead. I'm not really going anywhere with this yet, but I'm curious if anyone can see any paths that might lead to a simpler result? It's particularly bad at the moment, with me being UTC+11, Martin UTC+1, and Fred UTC-4 (I might be off by an hour or two for Fred and Martin). From bac at OCF.Berkeley.EDU Wed Nov 3 23:57:26 2004 From: bac at OCF.Berkeley.EDU (Brett C.) Date: Wed Nov 3 23:57:57 2004 Subject: [Python-Dev] TRUNK FROZEN for 2.4b2 release from 0000UTC 3rd November (tomorrrow) In-Reply-To: <41888D8E.5050705@interlink.com.au> References: <r01050400-1026-6FEC84FE2D6B11D9ADC9003065D5E7E4@[10.0.0.23]> <41888D8E.5050705@interlink.com.au> Message-ID: <41896256.1040104@ocf.berkeley.edu> Anthony Baxter wrote: > Just van Rossum wrote: > >> I don't get it. This is about an undocumented Mac-specific API function >> that noone uses (not even Python itself) which was introduced in 2.3. It >> just needs to be ripped out. I don't mind if it's done after b2, but I >> agree with Bob that it needs to be taken out before 2.4 final. Jack, >> would you agree? > I think Bob also wanted to get the whole patch in that removed compiling the core against CoreFoundation and CoreServices. I am not comfortable doing that without more lead into a beta than this so I am holding off until 2.5 for that. > > FWIW, I don't mind having this done either way... > OK, then I will just rip it out and skip the deprecation warning. -Brett From bac at OCF.Berkeley.EDU Thu Nov 4 00:13:34 2004 From: bac at OCF.Berkeley.EDU (Brett C.) Date: Thu Nov 4 00:13:54 2004 Subject: [Python-Dev] interlocking dependencies on the path to a release In-Reply-To: <4188BE21.6040205@interlink.com.au> References: <4188BE21.6040205@interlink.com.au> Message-ID: <4189661E.7040002@ocf.berkeley.edu> Anthony Baxter wrote: > So I've been documenting the current state of the release process, > and it's a bit messier than I'd like - particularly with people > scattered across timezones. Here's the list of things that need > to happen, and what depends on each. > > 1. Include/patchlevel.h > 2. Doc build (depends on 1.) > 3. Misc/NEWS, Lib/idlelib/NEWS.txt, Lib/idlelib/idlever.py > 4. PCbuild/BUILDno.txt, pythoncore.dsp > 5. Tag the tree (depends on 1, 3, 5) > 6. export/build the tarballs (depends on 5) > 7. Build the windows release (depends on 2, 5) > 8. Update the webpages, sign the files (depends on 6, 7) > 9. Send the email (depends on 8) > The only thing I can see possibly making things easier is to script stuff. Steps 1, 3, and 4 could all be automated. And if generating the HTML for the reST text is a slight pain because of errors in the syntax we could try to check that more often or even designate that someone keeps random track that the syntax is good. Otherwise we could probably try to have some build script on creosote (web server for python.org) that tries to generate it nightly to catch errors early. Best solution I can think of is somehow come up with a way to remove the need for Martin to do 4 somehow. If that happens then Fred can do 1-2, Anthony can do 3-6, Martin does 7, and then Anthony finished up with 8-10. Without Martin doing 7-10 I don't see a good way to rework this. -Brett From bob at redivi.com Thu Nov 4 00:14:08 2004 From: bob at redivi.com (Bob Ippolito) Date: Thu Nov 4 00:14:18 2004 Subject: [Python-Dev] TRUNK FROZEN for 2.4b2 release from 0000UTC 3rd November (tomorrrow) In-Reply-To: <41896256.1040104@ocf.berkeley.edu> References: <r01050400-1026-6FEC84FE2D6B11D9ADC9003065D5E7E4@[10.0.0.23]> <41888D8E.5050705@interlink.com.au> <41896256.1040104@ocf.berkeley.edu> Message-ID: <103B4788-2DEE-11D9-8F1A-000A95BA5446@redivi.com> On Nov 3, 2004, at 17:57, Brett C. wrote: > Anthony Baxter wrote: >> Just van Rossum wrote: >>> I don't get it. This is about an undocumented Mac-specific API >>> function >>> that noone uses (not even Python itself) which was introduced in >>> 2.3. It >>> just needs to be ripped out. I don't mind if it's done after b2, but >>> I >>> agree with Bob that it needs to be taken out before 2.4 final. Jack, >>> would you agree? > > I think Bob also wanted to get the whole patch in that removed > compiling the core against CoreFoundation and CoreServices. I am not > comfortable doing that without more lead into a beta than this so I am > holding off until 2.5 for that. Argh! It's nothing crazy, it removes incorrect linker flags and delegates two Mac-specific API functions to where they belong. -bob From martin at v.loewis.de Thu Nov 4 00:17:59 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu Nov 4 00:17:53 2004 Subject: [Python-Dev] Re: interlocking dependencies on the path to a release In-Reply-To: <4188BE21.6040205@interlink.com.au> References: <4188BE21.6040205@interlink.com.au> Message-ID: <41896727.7030206@v.loewis.de> Anthony Baxter wrote: > 1. Include/patchlevel.h > 2. Doc build (depends on 1.) > 3. Misc/NEWS, Lib/idlelib/NEWS.txt, Lib/idlelib/idlever.py > 4. PCbuild/BUILDno.txt, pythoncore.dsp > 5. Tag the tree (depends on 1, 3, 5) > 6. export/build the tarballs (depends on 5) > 7. Build the windows release (depends on 2, 5) > 8. Update the webpages, sign the files (depends on 6, 7) > 9. Send the email (depends on 8) > > > Stage 2 is a Fred thing, stages 4 and 7 are Martin, most of the rest > are things I do, although Fred will often do 1 because he needs it > to do 2, and the timezone thing makes it easier for him to just do > it. It would be possible to automate the bumps to version numbers, and you don't have to be on Windows to do 4, anymore (as the project file is now a fairly stable XML file). Also, it would be possible to bump version numbers immediately after a snapshot (alpha, beta) release, rather than just before. I tend to forget that I have to do 4, and remember at earliest when you announce the freeze. > (*) The names 'Fred', 'Martin', and 'I' are just the names of the > current set of release people - feel free to substitute roles in > there instead. I think it should be possible to come up with a release process that combines Fred's and my role, wrt. documentation creation (I create the chm file); theoretically, whoever builds the documentation should be able to do so on Windows as well. Also, I think it would be good to eliminate Fred's role altogether: the documentation build process should be much more automatic, and less relying on third-party tools. I think aiming at rewriting ht2html in Python for 2.5 would be a worthwhile goal (I have dropped the idea that conversion to xml would be worthwhile). Then, the release manager could trivially build the documentation. For that to work, it is necessary that the documentation builds out of the box; for that to happen, it is necessary that a broad audience can build the documentation, so that problems are detected long before the release (i.e. short after a checkin - or better yet, before). For the PostScript/PDF release, one certainly needs TeX, but that is free software. The same could be said about the Windows release, except that we stand little chance of ever dropping the requirement for special, non-free tools (Windows, MSVC). However, anybody possessing these tools and owning a sufficiently powerful machine should be able to release Python. Regards, Martin From martin at v.loewis.de Thu Nov 4 00:20:22 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu Nov 4 00:20:16 2004 Subject: [Python-Dev] interlocking dependencies on the path to a release In-Reply-To: <4189661E.7040002@ocf.berkeley.edu> References: <4188BE21.6040205@interlink.com.au> <4189661E.7040002@ocf.berkeley.edu> Message-ID: <418967B6.4040106@v.loewis.de> Brett C. wrote: > Best solution I can think of is somehow come up with a way to remove the > need for Martin to do 4 somehow. This shouldn't be that bad as it currently is. *Anybody* could actually do this - there is no need to do it within the release process (except perhaps during a1). I don't see a problem with the CVS reporting numbers that wait for a release a few weeks. Regards, Martin From martin at v.loewis.de Thu Nov 4 00:26:59 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu Nov 4 00:26:55 2004 Subject: [Python-Dev] TRUNK FROZEN for 2.4b2 release from 0000UTC 3rd November (tomorrrow) In-Reply-To: <103B4788-2DEE-11D9-8F1A-000A95BA5446@redivi.com> References: <r01050400-1026-6FEC84FE2D6B11D9ADC9003065D5E7E4@[10.0.0.23]> <41888D8E.5050705@interlink.com.au> <41896256.1040104@ocf.berkeley.edu> <103B4788-2DEE-11D9-8F1A-000A95BA5446@redivi.com> Message-ID: <41896943.9000906@v.loewis.de> Bob Ippolito wrote: > Argh! It's nothing crazy, it removes incorrect linker flags and > delegates two Mac-specific API functions to where they belong. I think Brett's thinking is that if this was a serious problem, it should have been fixed months ago. The fact that nobody bothered pushing that change before 2.4b1 indicates that the problem is not serious. As an incompatible change that fixes a non-serious problem, it shouldn't be applied before 2.5, in accordance with PEP 5 (which says that an incompatible change must see a deprecation first). That said, the release manager has said what he thinks about the issue, so anybody with commit privileges please do what they think is needed (I personally don't care much enough beyond writing this message). Regards, Martin From anthony at python.org Thu Nov 4 00:50:50 2004 From: anthony at python.org (Anthony Baxter) Date: Thu Nov 4 00:46:30 2004 Subject: [Python-Dev] RELEASED Python 2.4, beta 2 Message-ID: <41896EDA.5020805@python.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On behalf of the Python development team and the Python community, I'm happy to announce the second beta of Python 2.4. Python 2.4b2 is a beta release. We'd greatly appreciate it if you could download it, kick the tires and let us know of any problems you find, but it is probably not suitable for production usage. ~ http://www.python.org/2.4 Notable changes in beta2 include another bug fix in the intersection of weakrefs and the garbage collector, time.strptime() correctly handling %U and %W arguments, and a fix for a problem of old source code being cached and shown in tracebacks. There's a number of other bugfixes as well, see either the highlights, the What's New in Python 2.4, or the detailed NEWS file -- all available from the Python 2.4 webpage. This is the second beta release of Python 2.4. Python 2.4 is now in the beta part of the release cycle - there should be few or no new features, merely bug fixes. From here, we plan to have a release candidate in a couple of weeks, and a final release at the start of December - but obviously this plan is subject to change, based on any problems that might crop up. Please log any problems you have with this release in the SourceForge bug tracker (noting that you're using 2.4b2): ~ http://sourceforge.net/bugs/?group_id=5470 Enjoy the new release, Anthony Anthony Baxter anthony@python.org Python Release Manager (on behalf of the entire python-dev team) -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.6 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFBiW7YDt3F8mpFyBYRAq7DAKCskVfPO4LEQlC3zijupOnd9snCWACgpMHX bH7xepODsyo1115TMQ9CQkw= =0UOq -----END PGP SIGNATURE----- From anthony at interlink.com.au Thu Nov 4 00:52:31 2004 From: anthony at interlink.com.au (Anthony Baxter) Date: Thu Nov 4 00:48:14 2004 Subject: [Python-Dev] TRUNK FROZEN for 2.4b2 release from 0000UTC 3rd November (tomorrrow) In-Reply-To: <103B4788-2DEE-11D9-8F1A-000A95BA5446@redivi.com> References: <r01050400-1026-6FEC84FE2D6B11D9ADC9003065D5E7E4@[10.0.0.23]> <41888D8E.5050705@interlink.com.au> <41896256.1040104@ocf.berkeley.edu> <103B4788-2DEE-11D9-8F1A-000A95BA5446@redivi.com> Message-ID: <41896F3F.8090803@interlink.com.au> Bob Ippolito wrote: > Argh! It's nothing crazy, it removes incorrect linker flags and > delegates two Mac-specific API functions to where they belong. I asked this question on #macpython, but didn't get a response - I understand the costs for removing this are slight, but I'm not sure what the benefits of not linking the frameworks in are? Anyone? From bob at redivi.com Thu Nov 4 01:03:56 2004 From: bob at redivi.com (Bob Ippolito) Date: Thu Nov 4 01:04:03 2004 Subject: [Python-Dev] TRUNK FROZEN for 2.4b2 release from 0000UTC 3rd November (tomorrrow) In-Reply-To: <41896943.9000906@v.loewis.de> References: <r01050400-1026-6FEC84FE2D6B11D9ADC9003065D5E7E4@[10.0.0.23]> <41888D8E.5050705@interlink.com.au> <41896256.1040104@ocf.berkeley.edu> <103B4788-2DEE-11D9-8F1A-000A95BA5446@redivi.com> <41896943.9000906@v.loewis.de> Message-ID: <051E8585-2DF5-11D9-8F1A-000A95BA5446@redivi.com> On Nov 3, 2004, at 18:26, Martin v. L?wis wrote: > Bob Ippolito wrote: >> Argh! It's nothing crazy, it removes incorrect linker flags and >> delegates two Mac-specific API functions to where they belong. > > I think Brett's thinking is that if this was a serious problem, > it should have been fixed months ago. The fact that nobody bothered > pushing that change before 2.4b1 indicates that the problem is not > serious. As an incompatible change that fixes a non-serious problem, > it shouldn't be applied before 2.5, in accordance with PEP 5 > (which says that an incompatible change must see a deprecation first). The thing is that the incompatible change is going in, but the compatible changes (the reason for the incompatible change) are not. This is very peculiar. -bob From bob at redivi.com Thu Nov 4 01:14:31 2004 From: bob at redivi.com (Bob Ippolito) Date: Thu Nov 4 01:14:40 2004 Subject: [Python-Dev] TRUNK FROZEN for 2.4b2 release from 0000UTC 3rd November (tomorrrow) In-Reply-To: <41896F3F.8090803@interlink.com.au> References: <r01050400-1026-6FEC84FE2D6B11D9ADC9003065D5E7E4@[10.0.0.23]> <41888D8E.5050705@interlink.com.au> <41896256.1040104@ocf.berkeley.edu> <103B4788-2DEE-11D9-8F1A-000A95BA5446@redivi.com> <41896F3F.8090803@interlink.com.au> Message-ID: <80269A7A-2DF6-11D9-8F1A-000A95BA5446@redivi.com> On Nov 3, 2004, at 18:52, Anthony Baxter wrote: > Bob Ippolito wrote: >> Argh! It's nothing crazy, it removes incorrect linker flags and >> delegates two Mac-specific API functions to where they belong. > > I asked this question on #macpython, but didn't get a response - I > understand the costs for removing this are slight, but I'm not sure > what the benefits of not linking the frameworks in are? Anyone? I don't see your question in my backlog of #macpython, my client must have been disconnected at that time? It should shrink the memory footprint of Python and cause it to start up a bit faster (unsure if this difference is going to be very measurable). It might also produce a binary that's compatible with both Darwin and Mac OS X (untested, I have no Darwin installation). -bob From Scott.Daniels at Acm.Org Thu Nov 4 01:33:02 2004 From: Scott.Daniels at Acm.Org (Scott David Daniels) Date: Thu Nov 4 01:31:49 2004 Subject: [Python-Dev] Re: RELEASED Python 2.4, beta 2 In-Reply-To: <41896EDA.5020805@python.org> References: <41896EDA.5020805@python.org> Message-ID: <cmbt9d$ecu$1@sea.gmane.org> Anthony Baxter wrote: > On behalf of the Python development team and the Python community, I'm > happy to announce the second beta of Python 2.4. Congratulations, another gold star. -- -- Scott David Daniels Scott.Daniels@Acm.Org From jcarlson at uci.edu Thu Nov 4 01:52:27 2004 From: jcarlson at uci.edu (Josiah Carlson) Date: Thu Nov 4 02:00:45 2004 Subject: [Python-Dev] previously discussed binhex additions Message-ID: <20041103160824.F7F2.JCARLSON@uci.edu> Since it looks as though we're coming down to the wire on Python 2.4, I thought I would mention the previously discussed binhex additions. If desired, I would implement the long2bytes and bytes2long functions with documentation and test cases as described a month ago by Tim Peters: http://mail.python.org/pipermail/python-dev/2004-October/049206.html I could have it done this Saturday evening for testing/checking during Sunday's bug day. - Josiah From anthony at interlink.com.au Thu Nov 4 04:13:33 2004 From: anthony at interlink.com.au (Anthony Baxter) Date: Thu Nov 4 04:13:58 2004 Subject: [Python-Dev] Re: PEP 336, "Make None Callable", by Andrew McClelland In-Reply-To: <E1CPOSu-0005MB-RF@sc8-pr-cvs1.sourceforge.net> References: <E1CPOSu-0005MB-RF@sc8-pr-cvs1.sourceforge.net> Message-ID: <41899E5D.2080303@interlink.com.au> > Abstract > > None should be a callable object that when called with any > arguments has no side effect and returns None. My response to this is simply "yuck". This is a hack, abd I don't think it's worthwhile. If you want to test for None, test for None. Similarly, if you want a dictionary lookup to default to a no-op method, there's a perfectly simple way to spell it now: def defaultNoopMethod(*args, **kwargs): pass somedict.get(key, defaultNoopMethod) You can even do this as a one-liner using a lambda if you want. Anthony -- Anthony Baxter <anthony@interlink.com.au> It's never too late to have a happy childhood. From tim.peters at gmail.com Thu Nov 4 04:25:41 2004 From: tim.peters at gmail.com (Tim Peters) Date: Thu Nov 4 04:25:43 2004 Subject: [Python-Dev] Re: PEP 336, "Make None Callable", by Andrew McClelland In-Reply-To: <41899E5D.2080303@interlink.com.au> References: <E1CPOSu-0005MB-RF@sc8-pr-cvs1.sourceforge.net> <41899E5D.2080303@interlink.com.au> Message-ID: <1f7befae0411031925522b6bb@mail.gmail.com> [Anthony Baxter] >> Abstract >> >> None should be a callable object that when called with any >> arguments has no side effect and returns None. > My response to this is simply "yuck". This is a hack, abd I > don't think it's worthwhile. Maybe it's not a coincidence that it's listed in the PEP index next to "Reject Foolish Indentation" <0.9 wink>. IMO, it's worse than not worthwhile: it would be actively bad to implement this. TypeError: 'NoneType' object is not callable is a frequent runtime occurrence now, and performs a valuable service when it happens by correctly pointing out a programming error. Python's None isn't meant to be neutral -- it's meant to mean "nothing is there", and supplying nothing where something is required is a logic error. From anthony at interlink.com.au Thu Nov 4 04:29:37 2004 From: anthony at interlink.com.au (Anthony Baxter) Date: Thu Nov 4 04:42:12 2004 Subject: [Python-Dev] TRUNK UNFROZEN Message-ID: <4189A221.4020306@interlink.com.au> The trunk is open again for checkins. As before, we're in feature-freeze mode. Next release, which will hopefully be 2.4rc1, in 2 weeks time. -- Anthony Baxter <anthony@interlink.com.au> It's never too late to have a happy childhood. From bac at OCF.Berkeley.EDU Thu Nov 4 05:24:53 2004 From: bac at OCF.Berkeley.EDU (Brett C.) Date: Thu Nov 4 05:25:11 2004 Subject: [Python-Dev] TRUNK FROZEN for 2.4b2 release from 0000UTC 3rd November (tomorrrow) In-Reply-To: <051E8585-2DF5-11D9-8F1A-000A95BA5446@redivi.com> References: <r01050400-1026-6FEC84FE2D6B11D9ADC9003065D5E7E4@[10.0.0.23]> <41888D8E.5050705@interlink.com.au> <41896256.1040104@ocf.berkeley.edu> <103B4788-2DEE-11D9-8F1A-000A95BA5446@redivi.com> <41896943.9000906@v.loewis.de> <051E8585-2DF5-11D9-8F1A-000A95BA5446@redivi.com> Message-ID: <4189AF15.9060407@ocf.berkeley.edu> Bob Ippolito wrote: > > On Nov 3, 2004, at 18:26, Martin v. L?wis wrote: > >> Bob Ippolito wrote: >> >>> Argh! It's nothing crazy, it removes incorrect linker flags and >>> delegates two Mac-specific API functions to where they belong. >> >> >> I think Brett's thinking is that if this was a serious problem, >> it should have been fixed months ago. The fact that nobody bothered >> pushing that change before 2.4b1 indicates that the problem is not >> serious. As an incompatible change that fixes a non-serious problem, >> it shouldn't be applied before 2.5, in accordance with PEP 5 >> (which says that an incompatible change must see a deprecation first). > Exactly. It's too late in the release cycle for me to put it in unless Anthony says it is okay. > > The thing is that the incompatible change is going in, but the > compatible changes (the reason for the incompatible change) are not. > This is very peculiar. > Not really. Only one function is being removed. The function rearrangement is not under discussion here. Plus there is the issue of testing. I obviously checked it out and it seems to work, but this is build stuff which can be touchy. If Anthony clears applying all the full patch I will, but otherwise I am playing it safe and waiting until 2.5 . -Brett From python at rcn.com Thu Nov 4 05:54:14 2004 From: python at rcn.com (Raymond Hettinger) Date: Thu Nov 4 05:57:08 2004 Subject: [Python-Dev] previously discussed binhex additions In-Reply-To: <20041103160824.F7F2.JCARLSON@uci.edu> Message-ID: <002801c4c22a$6e4a6200$e841fea9@oemcomputer> > Since it looks as though we're coming down to the wire on Python 2.4, I > thought I would mention the previously discussed binhex additions. Too late for Py2.4. Bugfixes only. Not too early to submit a Py2.5 patch. Raymond From anthony at interlink.com.au Thu Nov 4 05:58:40 2004 From: anthony at interlink.com.au (Anthony Baxter) Date: Thu Nov 4 05:59:14 2004 Subject: [Python-Dev] TRUNK FROZEN for 2.4b2 release from 0000UTC 3rd November (tomorrrow) In-Reply-To: <4189AF15.9060407@ocf.berkeley.edu> References: <r01050400-1026-6FEC84FE2D6B11D9ADC9003065D5E7E4@[10.0.0.23]> <41888D8E.5050705@interlink.com.au> <41896256.1040104@ocf.berkeley.edu> <103B4788-2DEE-11D9-8F1A-000A95BA5446@redivi.com> <41896943.9000906@v.loewis.de> <051E8585-2DF5-11D9-8F1A-000 Message-ID: <4189B700.4010004@interlink.com.au> Brett C. wrote: > Not really. Only one function is being removed. The function > rearrangement is not under discussion here. Plus there is the issue of > testing. I obviously checked it out and it seems to work, but this is > build stuff which can be touchy. > > If Anthony clears applying all the full patch I will, but otherwise I am > playing it safe and waiting until 2.5 . Bear in mind that we've had _no_ macpython releases in the 2.4 cycle so far, that I'm aware of. This is really a platform-specific build issue. From python at rcn.com Thu Nov 4 06:06:10 2004 From: python at rcn.com (Raymond Hettinger) Date: Thu Nov 4 06:09:12 2004 Subject: [Python-Dev] TRUNK FROZEN for 2.4b2 release from 0000UTC 3rdNovember (tomorrrow) In-Reply-To: <051E8585-2DF5-11D9-8F1A-000A95BA5446@redivi.com> Message-ID: <000001c4c22c$1daf6500$e841fea9@oemcomputer> [Bob Ippolito] > The thing is that the incompatible change is going in, but the > compatible changes (the reason for the incompatible change) are not. > This is very peculiar. [Anthony] > Bear in mind that we've had _no_ macpython releases in the 2.4 cycle > so far, that I'm aware of. This is really a platform-specific build > issue. Poor Bob, this reasonable request has been kicked around too much. It is a simple thing that strips unnecessary stuff out of the build. Will someone who has a Mac please apply. Raymond From tim.peters at gmail.com Thu Nov 4 06:25:32 2004 From: tim.peters at gmail.com (Tim Peters) Date: Thu Nov 4 06:25:35 2004 Subject: [Python-Dev] Re: weakref gc semantics In-Reply-To: <1f7befae04110220402077a6fa@mail.gmail.com> References: <1f7befae04110220402077a6fa@mail.gmail.com> Message-ID: <1f7befae041103212576c59c42@mail.gmail.com> [Tim] > This example bothers me a little, under current Python CVS: > > """ > import gc, weakref > > def C_gone(ignored): > print "An object of type C went away." > > class C: > def __init__(self): > self.wr = weakref.ref(self, C_gone) > > print "creating non-cyclic C instance" > c = C() > > print "creating cyclic C instance and destroying old one" > c = C() # triggers the print > c.loop = c > > print "getting rid of the cyclic one" > c = None # doesn't trigger the print > gc.collect() # ditto > """ Recapping, it bothers me because the callback doesn't trigger now when c is part of cyclic trash, but if c.wr were an instance with a __del__ method instead, the __del__ method *would* trigger. There are (at least) two easy-to-code and pretty-easy-to-explain ways to get that callback invoked safely. One I sketched last time, and is in the attached patch-callback.txt. (Patches here just have gcmodule.c code changes, no corresponding changes to comments.) The difference is that it invokes a callback on a weakref to trash iff the callback is reachable, instead of invoking a callback on a weakref to trash iff the weakref it's attached to is reachable. All the tests we've accumulated still pass with this change, and the example above does trigger a second print. Another approach is in the attached patch-finalizer.txt. This is more (I think) like what Neil was aiming at last week, and amounts to changing has_finalizer() to return true when it sees a weakref with a callback. Then the weakref, and everything reachable from it, gets moved to the reachable set. After that, no callback can access an unreachable object directly, so callbacks are safe to call on that count. We still (and in the other patch too) clear all weakrefs to trash before breaking cycles, so callbacks can't resurrect trash via active weakrefs either. But in the patch-finalizer case, the callbacks attached to weakrefs moved to the reachable set no longer get invoked "artificially" (forced by gc) -- they trigger iff their referents eventually go away. Some current tests fail with the patch-finalizer approach, but for shallow reasons: fewer callbacks get invoked, and some of the tests currently use "global" (reachable) weakref callbacks as a way to verify that gc really did clean up cyclic trash. Those stop getting invoked. Some of the tests use bound method objects as callbacks, and when a weakref contains a callback that's a bound method object, and the weakref gets moved to the reachable set, its callback gets moved there too, and so the instance wrt which the callback is a bound method becomes reachable too, and so the class of that instance also becomes reachable. As a result, gc doesn't get rid of any of that stuff, so the "external" callbacks never trigger. Instead, the weakrefs with the bound-method callbacks end up in gc.garbage. I don't like that, because if it's possible to get rid of __del__ methods, I'd really like to see gc.garbage go away too. Nothing we've done with weakrefs so far can add anything to gc.garbage, so I like them better on that count. Pragmatically, sticking a weakref in gc.garbage isn't of much use to the end-user now either, because a weakref's callback is neither clearable nor even discoverable now at the Python level. But that could be changed too. What I'm missing is a pile of practical <heh> use cases to distinguish these. Let's look at weak value dicts, for reasons that will become clear. An entry in a WVD d now is of the form key: W where W is an instance of a subclass of weakref.ref, and W has the associated key as an attribute. Every value in d shares the same callback, set up by WVD.__init__(): def __init__(self, *args, **kw): ... def remove(wr, selfref=ref(self)): self = selfref() if self is not None: del self.data[wr.key] self._remove = remove So the callback has a weakref to d, and when it's invoked extracts the specific key from the weakref instance and uses that to delete the appropriate key: W entry from d via the shared callback's own weakref to d. Now it's not clear *why* the callback has a weakref to d. It *could* have been written like this instead: def remove(wr): del self.data[wr.key] In fact, at one time it was written in that kind of way. Rev 1.13 of weakref.py changed it to use a weakref to self, and the motivations in the patch that started this are feeble: http://www.python.org/sf/417795 "It's bad to rely on the garbage collector" and "this could lead to uncollectable cycles if a subclass of Weak*Dictionary defines __del__". The proper responses were "no, it isn't" and "insane code deserves whatever it gets" <0.5 wink>. Anyway, if WVD *had* been left simple(r), it would be relevant to the choices above: if WVD d became part of cyclic trash, patch-finalizer would move all its weakrefs to the reachable set, and then d itself would be reachable too (via the shared callback's strong reference to self). As a result, none of the callbacks would trigger, and d would "leak" (it wouldn't get cleaned up, and wouldn't show up in gc.garbage either -- it would end up in an older generation, clogging gc forever after because it would remain forever after reachable from its (resurrected) weakrefs in gc.garbage). Under patch-callback, d would still go away (the simplest way to understand why is just that d *is* in cyclic trash, and patch-callback never moves anything to the reachable set because of weakrefs or weakref callbacks). The *current* implementation of WVD doesn't have this problem under patch-finalizer. If d is part of cyclic trash, all its weakrefs get moved to the reachable set, but d isn't directly reachable from their shared callback, so d is not moved to the reachable set. d's tp_clear() gets invoked by gc in its cycle-breaking phase, and that decrefs all of d's weakrefs that were moved to the unreachable set, and they go away then (without triggering their callbacks) via the normal refcount route. The point is that this is all pretty subtle, and it bothers me that "the obvious" way to implement WVD (the pre-patch-417795 way) would lead to a worse outcome under patch-finalizer, so I also fear that Python's current weakref users may be doing similar things now that would also suffer under patch-finalizer. OTOH, there's something conceptually nice about patch-finalizer -- it is the easiest approach to explain (well, except for approaches that allow segfaults to occur ...). On the third hand, gc.garbage appears poorly understood by users now, and patch-finalizer is actually easy to explain only to those who already understand gc.garbage. -------------- next part -------------- Index: Modules/gcmodule.c =================================================================== RCS file: /cvsroot/python/python/dist/src/Modules/gcmodule.c,v retrieving revision 2.80 diff -u -r2.80 gcmodule.c --- Modules/gcmodule.c 1 Nov 2004 16:39:57 -0000 2.80 +++ Modules/gcmodule.c 4 Nov 2004 03:37:50 -0000 @@ -566,9 +566,8 @@ * to imagine how calling it later could create a problem for us. wr * is moved to wrcb_to_call in this case. */ - if (IS_TENTATIVELY_UNREACHABLE(wr)) + if (IS_TENTATIVELY_UNREACHABLE(wr->wr_callback)) continue; - assert(IS_REACHABLE(wr)); /* Create a new reference so that wr can't go away * before we can process it again. @@ -577,9 +576,8 @@ /* Move wr to wrcb_to_call, for the next pass. */ wrasgc = AS_GC(wr); - assert(wrasgc != next); /* wrasgc is reachable, but - next isn't, so they can't - be the same */ + if (wrasgc == next) + next = wrasgc->gc.gc_next; gc_list_move(wrasgc, &wrcb_to_call); } } @@ -593,7 +591,6 @@ gc = wrcb_to_call.gc.gc_next; op = FROM_GC(gc); - assert(IS_REACHABLE(op)); assert(PyWeakref_Check(op)); wr = (PyWeakReference *)op; callback = wr->wr_callback; @@ -621,6 +618,7 @@ if (wrcb_to_call.gc.gc_next == gc) { /* object is still alive -- move it */ gc_list_move(gc, old); + gc->gc.gc_refs = GC_REACHABLE; } else ++num_freed; -------------- next part -------------- Index: Modules/gcmodule.c =================================================================== RCS file: /cvsroot/python/python/dist/src/Modules/gcmodule.c,v retrieving revision 2.80 diff -u -r2.80 gcmodule.c --- Modules/gcmodule.c 1 Nov 2004 16:39:57 -0000 2.80 +++ Modules/gcmodule.c 4 Nov 2004 03:27:10 -0000 @@ -413,6 +413,9 @@ assert(delstr != NULL); return _PyInstance_Lookup(op, delstr) != NULL; } + else if (PyWeakref_Check(op) && + ((PyWeakReference *)op)->wr_callback != NULL) + return 1; else if (PyType_HasFeature(op->ob_type, Py_TPFLAGS_HEAPTYPE)) return op->ob_type->tp_del != NULL; else @@ -566,10 +569,6 @@ * to imagine how calling it later could create a problem for us. wr * is moved to wrcb_to_call in this case. */ - if (IS_TENTATIVELY_UNREACHABLE(wr)) - continue; - assert(IS_REACHABLE(wr)); - /* Create a new reference so that wr can't go away * before we can process it again. */ From jcarlson at uci.edu Thu Nov 4 09:43:07 2004 From: jcarlson at uci.edu (Josiah Carlson) Date: Thu Nov 4 09:51:01 2004 Subject: [Python-Dev] previously discussed binhex additions In-Reply-To: <002801c4c22a$6e4a6200$e841fea9@oemcomputer> References: <20041103160824.F7F2.JCARLSON@uci.edu> <002801c4c22a$6e4a6200$e841fea9@oemcomputer> Message-ID: <20041104004215.F809.JCARLSON@uci.edu> "Raymond Hettinger" <python@rcn.com> wrote: > > > Since it looks as though we're coming down to the wire on Python 2.4, > I > > thought I would mention the previously discussed binhex additions. > > Too late for Py2.4. > Bugfixes only. > Not too early to submit a Py2.5 patch. I'll wait until 2.4 is released then. If there's no rush, then I guess I'm not in one. - Josiah From FBatista at uniFON.com.ar Thu Nov 4 13:48:34 2004 From: FBatista at uniFON.com.ar (Batista, Facundo) Date: Thu Nov 4 13:50:13 2004 Subject: [Python-Dev] Policy about old Python versions Message-ID: <A128D751272CD411BC9200508BC2194D053C7B5D@escpl.tcp.com.ar> Sometimes, reviewing bugs, I found some ones referring problems about old Python versions. Is there a policy about supporting old Python versions? At some time an old version becomes "deprecated"? For example, an open bug about a problem of Python 2.1, that is solved in Python 2.2 and subsequents, when should be closed? Thank you! . Facundo From walter at livinglogic.de Thu Nov 4 13:54:51 2004 From: walter at livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=) Date: Thu Nov 4 13:54:56 2004 Subject: [Python-Dev] Code coverage tool updated In-Reply-To: <1f7befae04110210537c35e514@mail.gmail.com> References: <41864572.4000806@livinglogic.de> <16774.45652.835103.327656@montanaro.dyndns.org> <4187765D.3000705@livinglogic.de> <4187CA1C.9050508@livinglogic.de> <1f7befae04110210537c35e514@mail.gmail.com> Message-ID: <418A269B.6010207@livinglogic.de> Tim Peters wrote: > [Walter D?rwald] > >>This doesn't help. The following assert in trace.py raises >>an AssertionError: >> assert filename.endswith('.py') >> >>[...] >> >>So what is <doctest _threading_local[7]>? > > I haven't followed this thread, but can answer that question literally > <wink>: it's code synthesized for the seventh doctest example in > Lib/_threading_local.py's module docstring. The file name is > constructed by this line in doctest.py: > > # Use a special filename for compile(), so we can retrieve > # the source code during interactive debugging (see > # __patched_linecache_getlines). > filename = '<doctest %s[%d]>' % (test.name, examplenum) I guess it's not worth it to try to fix doctest/trace to provide sourcecode for doctest code, but IMHO trace should be able to survive a doctest. Removing the assert statement, so that trace.py runs to completion shows a different problem: There are only 32 files covered according to the trace output. The complete test log can be found here: http://styx.livinglogic.de/~walter/brokentrace.txt The trace call looked like this: ./python ../../../trace.py --count --summary --missing Lib/test/regrtest.py with ../../../trace.py being the trace.py from current CVS with the assert statement removed. So am I doing something wrong or is trace.py broken? Bye, Walter D?rwald From aahz at pythoncraft.com Thu Nov 4 15:16:32 2004 From: aahz at pythoncraft.com (Aahz) Date: Thu Nov 4 15:16:35 2004 Subject: [Python-Dev] Policy about old Python versions In-Reply-To: <A128D751272CD411BC9200508BC2194D053C7B5D@escpl.tcp.com.ar> References: <A128D751272CD411BC9200508BC2194D053C7B5D@escpl.tcp.com.ar> Message-ID: <20041104141632.GA27459@panix.com> On Thu, Nov 04, 2004, Batista, Facundo wrote: > > Sometimes, reviewing bugs, I found some ones referring problems about > old Python versions. > > Is there a policy about supporting old Python versions? At some time > an old version becomes "deprecated"? > > For example, an open bug about a problem of Python 2.1, that is solved > in Python 2.2 and subsequents, when should be closed? As soon as the new version is released, if there is no bugfix planned for that issue. Two versions later (i.e. 2.3 in this case) even if it looks like it ought to be fixed in 2.1 -- we haven't been doing bugfixes more than one version old. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ WiFi is the SCSI of the 21st Century -- there are fundamental technical reasons for sacrificing a goat. (with no apologies to John Woods) From misa at redhat.com Thu Nov 4 16:38:19 2004 From: misa at redhat.com (Mihai Ibanescu) Date: Thu Nov 4 16:38:23 2004 Subject: [Python-Dev] Overflow in socketmodule.c? Message-ID: <20041104153819.GS13571@abulafia.devel.redhat.com> Hello, Can someone confirm this is indeed an overflow by one in socketmodule.c? static PyObject * socket_inet_ntop(PyObject *self, PyObject *args) { int af; char* packed; int len; const char* retval; #ifdef ENABLE_IPV6 char ip[MAX(INET_ADDRSTRLEN, INET6_ADDRSTRLEN) + 1]; #else char ip[INET_ADDRSTRLEN + 1]; #endif /* Guarantee NUL-termination for PyString_FromString() below */ memset((void *) &ip[0], '\0', sizeof(ip) + 1); If it is I'll go ahead and file it. Thanks, Misa From gjc at inescporto.pt Thu Nov 4 16:56:07 2004 From: gjc at inescporto.pt (Gustavo J. A. M. Carneiro) Date: Thu Nov 4 16:56:30 2004 Subject: [Python-Dev] Overflow in socketmodule.c? In-Reply-To: <20041104153819.GS13571@abulafia.devel.redhat.com> References: <20041104153819.GS13571@abulafia.devel.redhat.com> Message-ID: <1099583767.31972.15.camel@spectrum.inescn.pt> Qui, 2004-11-04 ?s 10:38 -0500, Mihai Ibanescu escreveu: > Hello, > > Can someone confirm this is indeed an overflow by one in socketmodule.c? > > > static PyObject * > socket_inet_ntop(PyObject *self, PyObject *args) > { > int af; > char* packed; > int len; > const char* retval; > #ifdef ENABLE_IPV6 > char ip[MAX(INET_ADDRSTRLEN, INET6_ADDRSTRLEN) + 1]; > #else > char ip[INET_ADDRSTRLEN + 1]; > #endif > > /* Guarantee NUL-termination for PyString_FromString() below */ > memset((void *) &ip[0], '\0', sizeof(ip) + 1); > > > If it is I'll go ahead and file it. Indeed, looks like buffer overflow to me.. > > Thanks, > Misa > _______________________________________________ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/gjc%40inescporto.pt -- Gustavo J. A. M. Carneiro <gjc@inescporto.pt> <gustavo@users.sourceforge.net> The universe is always one step beyond logic. -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 3086 bytes Desc: not available Url : http://mail.python.org/pipermail/python-dev/attachments/20041104/2ddbd9ea/smime.bin From tim at zope.com Thu Nov 4 18:19:54 2004 From: tim at zope.com (Tim Peters) Date: Thu Nov 4 18:20:01 2004 Subject: [Python-Dev] patch-finalizer vs Zope3 Message-ID: <20041104171957.A0C633B8038@smtp.zope.com> FYI, and without taking time for non-trivial analysis. I ran the non-ZEO Zope3 tests with a Python patched to treat weakrefs with callbacks as reachable (this was patch-finalizer.txt, attached to a msg I sent to python-dev last night). The first test to leave stuff in gc.garbage then: The following test left garbage: test_created (bugtracker.browser.tests.test_bug.BugBaseViewTest) 65 where 65 is len(gc.garbage) after the test finished. From there it climbed steadily. By the time the functional tests ended, the Python process was about 575MB, and gc.garbage contained 324,056 weakrefs. I haven't investigated why it happens, but that it *does* happen in an app that believes it's well-behaved suggests that "practicality beats purity" would win in the end (i.e., no matter how elegant the approach, if it's too hard to live with, it won't fly). Of course nothing is left in gc.garbage here under 2.3.4 or 2.4b2, or under 2.4b2 + patch-callback. Under the latter, the Python process tops out at about 150MB. These were all with debug builds of Python, so process sizes are substantially larger than with release builds (the object header is 2 pointers bigger in a debug build, and every object gotten via pymalloc grows 16 debug pad bytes, and everything gotten via PyMem_XYZ() gets pymalloc debug padding too). From jim at zope.com Thu Nov 4 18:24:53 2004 From: jim at zope.com (Jim Fulton) Date: Thu Nov 4 18:24:57 2004 Subject: [Python-Dev] Re: patch-finalizer vs Zope3 In-Reply-To: <20041104171957.A0C633B8038@smtp.zope.com> References: <20041104171957.A0C633B8038@smtp.zope.com> Message-ID: <418A65E5.6070403@zope.com> Tim Peters wrote: > FYI, and without taking time for non-trivial analysis. I ran the non-ZEO > Zope3 tests with a Python patched to treat weakrefs with callbacks as > reachable (this was patch-finalizer.txt, attached to a msg I sent to > python-dev last night). > > The first test to leave stuff in gc.garbage then: > > The following test left garbage: > test_created (bugtracker.browser.tests.test_bug.BugBaseViewTest) > 65 > > where 65 is len(gc.garbage) after the test finished. From there it climbed > steadily. By the time the functional tests ended, the Python process was > about 575MB, and gc.garbage contained 324,056 weakrefs. > > I haven't investigated why it happens, but that it *does* happen in an app > that believes it's well-behaved It's all a question of your definition of good behavior. ;) > suggests that "practicality beats purity" > would win in the end (i.e., no matter how elegant the approach, if it's too > hard to live with, it won't fly). Treating weakrefs with callbacks as reachable is a *big* change, though in subtle ways. I agree that this is too big for 2.4, but I *do* think it should be considered for 2.5. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From nas at arctrix.com Thu Nov 4 18:38:35 2004 From: nas at arctrix.com (Neil Schemenauer) Date: Thu Nov 4 18:38:39 2004 Subject: [Python-Dev] Re: weakref gc semantics In-Reply-To: <1f7befae041103212576c59c42@mail.gmail.com> References: <1f7befae04110220402077a6fa@mail.gmail.com> <1f7befae041103212576c59c42@mail.gmail.com> Message-ID: <20041104173835.GA15004@mems-exchange.org> On Thu, Nov 04, 2004 at 12:25:32AM -0500, Tim Peters wrote: > One I sketched last time, and is in the attached patch-callback.txt. > (Patches here just have gcmodule.c code changes, no corresponding > changes to comments.) The difference is that it invokes a callback on > a weakref to trash iff the callback is reachable, instead of invoking > a callback on a weakref to trash iff the weakref it's attached to is > reachable. I like this idea best. I don't see why it should be hard to explain. Instead of saying: If you want the weakref callback to be invoked, ensure the WEAKREF outlives the referent. we say: If you want the weakref callback to be invoked, ensure the CALLBACK outlives the referent. I think there may be one small problem though. Callbacks get passed the weakref object. Doesn't that break the rule that trash should not be exposed to Python code while the collection is taking place? Neil From jim at zope.com Thu Nov 4 18:47:00 2004 From: jim at zope.com (Jim Fulton) Date: Thu Nov 4 18:47:08 2004 Subject: [Python-Dev] Re: weakref gc semantics In-Reply-To: <20041104173835.GA15004@mems-exchange.org> References: <1f7befae04110220402077a6fa@mail.gmail.com> <1f7befae041103212576c59c42@mail.gmail.com> <20041104173835.GA15004@mems-exchange.org> Message-ID: <418A6B14.8080206@zope.com> Neil Schemenauer wrote: > On Thu, Nov 04, 2004 at 12:25:32AM -0500, Tim Peters wrote: > >>One I sketched last time, and is in the attached patch-callback.txt. >>(Patches here just have gcmodule.c code changes, no corresponding >>changes to comments.) The difference is that it invokes a callback on >>a weakref to trash iff the callback is reachable, instead of invoking >>a callback on a weakref to trash iff the weakref it's attached to is >>reachable. > > > I like this idea best. I don't see why it should be hard to > explain. Instead of saying: > > If you want the weakref callback to be invoked, ensure the > WEAKREF outlives the referent. > > we say: > > If you want the weakref callback to be invoked, ensure the > CALLBACK outlives the referent. > > I think there may be one small problem though. Callbacks get passed > the weakref object. Doesn't that break the rule that trash should > not be exposed to Python code while the collection is taking place? Exactly, that's why I prefer the other approach. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From misa at redhat.com Thu Nov 4 18:47:26 2004 From: misa at redhat.com (Mihai Ibanescu) Date: Thu Nov 4 18:47:29 2004 Subject: [Python-Dev] Overflow in socketmodule.c? In-Reply-To: <1099583767.31972.15.camel@spectrum.inescn.pt> References: <20041104153819.GS13571@abulafia.devel.redhat.com> <1099583767.31972.15.camel@spectrum.inescn.pt> Message-ID: <20041104174726.GT13571@abulafia.devel.redhat.com> On Thu, Nov 04, 2004 at 03:56:07PM +0000, Gustavo J. A. M. Carneiro wrote: > Qui, 2004-11-04 ?s 10:38 -0500, Mihai Ibanescu escreveu: > > Hello, > > > > Can someone confirm this is indeed an overflow by one in socketmodule.c? > > > > > > static PyObject * > > socket_inet_ntop(PyObject *self, PyObject *args) > > { > > int af; > > char* packed; > > int len; > > const char* retval; > > #ifdef ENABLE_IPV6 > > char ip[MAX(INET_ADDRSTRLEN, INET6_ADDRSTRLEN) + 1]; > > #else > > char ip[INET_ADDRSTRLEN + 1]; > > #endif > > > > /* Guarantee NUL-termination for PyString_FromString() below */ > > memset((void *) &ip[0], '\0', sizeof(ip) + 1); > > > > > > If it is I'll go ahead and file it. > > Indeed, looks like buffer overflow to me.. Filed as SF bug 105470 Misa From martin at v.loewis.de Thu Nov 4 18:51:56 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu Nov 4 18:51:48 2004 Subject: [Python-Dev] Policy about old Python versions In-Reply-To: <A128D751272CD411BC9200508BC2194D053C7B5D@escpl.tcp.com.ar> References: <A128D751272CD411BC9200508BC2194D053C7B5D@escpl.tcp.com.ar> Message-ID: <418A6C3C.3050008@v.loewis.de> Batista, Facundo wrote: > Sometimes, reviewing bugs, I found some ones referring problems about old > Python versions. > > Is there a policy about supporting old Python versions? At some time an old > version becomes "deprecated"? > > For example, an open bug about a problem of Python 2.1, that is solved in > Python 2.2 and subsequents, when should be closed? It is (IMO) fine to close anything now that is fixed in 2.3 and newer. The proper "Resolution" is "Fixed", with a Comment "Fixed in 2.3". There likely will be one more 2.3 release, so you might not want to close something as "Fixed in 2.4" that is still incorrect in 2.3; instead, consider backporting the patch (we are all volunteers, so this is clearly not mandatory). If you find something is fixed in 2.4, but have no time to find out how precisely it was fixed, still leave a comment saying so. After the next 2.3 release, people are still free to backport to older branches - after all, distributors may chose to roll out patches collected even after the last release of some 2.x. However, in general, expect that 2.3 and older needs no further attention RSN, and "backporting" will mean "to 2.4". Regards, Martin From martin at v.loewis.de Thu Nov 4 19:05:28 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu Nov 4 19:05:20 2004 Subject: [Python-Dev] Overflow in socketmodule.c? In-Reply-To: <20041104153819.GS13571@abulafia.devel.redhat.com> References: <20041104153819.GS13571@abulafia.devel.redhat.com> Message-ID: <418A6F68.8080304@v.loewis.de> Mihai Ibanescu wrote: > Can someone confirm this is indeed an overflow by one in socketmodule.c? Yes. With some luck (e.g. on x86 with gcc), it might not cause buffer overruns, as a) the compiler overallocates on the stack because of padding, anyway, and b) the overwriting might write into the next variable (i.e. retval), which is uninitialized at this point, anyway. Regards, Martin From tim at zope.com Thu Nov 4 19:38:38 2004 From: tim at zope.com (Tim Peters) Date: Thu Nov 4 19:38:43 2004 Subject: [Python-Dev] RE: patch-finalizer vs Zope3 In-Reply-To: <418A65E5.6070403@zope.com> Message-ID: <20041104183839.AABEF3B8038@smtp.zope.com> [Jim Fulton] > Treating weakrefs with callbacks as reachable is a *big* change, though > in subtle ways. It sure appears to have "more consequences" than I guessed it would. > I agree that this is too big for 2.4, but I *do* think it should be > considered for 2.5. Me too. It ensures that a callback is never ignored unless a referent definitely outlives a weakref with that callback, even at the cost of resurrecting trash weakrefs (to make their callbacks safe to call). "Definitely outlives" is usually easy to talk about in CPython because of refcount lifetime semantics, but hard to make sense of operationally without refcount semantics. When they're both in cyclic trash, a weakref and its referent both become unreachable truly at the same instant. The approach so far has then been to say (in effect): Fine, then it's legitimate to impose any linear order on death, and if we say the weakref dies first, then there's no issue remaining -- just as in non-cyclic trash, when a weakref dies before its referent, the wr's callback will never be invoked. That's defensible but not compelling. I'm still not sure any other approach is compelling either. patch-finalizer effectly reverses the current approach, saying: Fine, then it's legitimate to impose any linear order on death, and if we say the referent dies first, then there's no issue remaining -- just as in non-cyclic trash, when the referent dies before its weakref, the wr's callback will be invoked(*). (*) Except we have to resurrect the weakref to make this possible, and that may also end up resurrecting the referent, and then neither actually goes away, and so the callback doesn't actually get invoked either (although it may get invoked later, if the user fiddles with the stuff in gc.garbage enough to make that possible). All I can say is that the callback better be damned important to justify all the extra pain <wink>. still-seeking-use-cases-ly y'rs - tim From foom at fuhm.net Thu Nov 4 19:44:22 2004 From: foom at fuhm.net (James Y Knight) Date: Thu Nov 4 19:44:26 2004 Subject: [Python-Dev] weakref gc semantics In-Reply-To: <1f7befae04110220402077a6fa@mail.gmail.com> References: <1f7befae04110220402077a6fa@mail.gmail.com> Message-ID: <8AFE1902-2E91-11D9-AF2F-000A95A50FB2@fuhm.net> On Nov 2, 2004, at 11:40 PM, Tim Peters wrote: > a reachable callback can have visible effects, and there > really isn't ambiguity about which of c and W "dies first" (c does) in > the example. An unreachable callback function can also have visible effects. > The question is whether gc should really be looking at whether the > weakref *callback* is reachable, regardless of whether the weakref > itself is reachable (if the weakref is reachable, its callback is too, > and 2.4b2 invokes it -- the only case in question is the one in the > example, where the weakref and the weakref's referent are unreachable > but the weakref's callback is reachable). From what I'm hearing here, the following two pieces of code would then have different behavior under patch-callback: """ def C_gone(ignored): print "An object of type C went away." class C: def __init__(self): self.wr = weakref.ref(self, C_gone) """ and """ class C: def __init__(self): def C_gone(ignored): print "An object of type C went away." self.wr = weakref.ref(self, C_gone) """ If that's correct, it sounds like a bad idea to me. James From tim.peters at gmail.com Thu Nov 4 20:32:51 2004 From: tim.peters at gmail.com (Tim Peters) Date: Thu Nov 4 20:32:57 2004 Subject: [Python-Dev] Re: weakref gc semantics In-Reply-To: <20041104173835.GA15004@mems-exchange.org> References: <1f7befae04110220402077a6fa@mail.gmail.com> <1f7befae041103212576c59c42@mail.gmail.com> <20041104173835.GA15004@mems-exchange.org> Message-ID: <1f7befae04110411327af9bf3d@mail.gmail.com> [Neil Schemenauer] > I like this idea [patch-callback] best. I don't see why it should be hard to > explain. Instead of saying: > > If you want the weakref callback to be invoked, ensure the > WEAKREF outlives the referent. > > we say: > > If you want the weakref callback to be invoked, ensure the > CALLBACK outlives the referent. The former is accurate independent of whether we're talking about cyclic trash now, but that latter would not be. For example, def callback(ignore): print 'hi' class C: pass c = C() w = weakref.ref(c, callback) w = None c = None The callback outlives c, but the callback doesn't get invoked, because w didn't outlive c. The callback does get invoked if the order of the last two lines is swapped. It's easy to explain this in terms of which of {c, w} goes away first, but not in terms of callback's lifetime. > I think there may be one small problem though. Callbacks get passed > the weakref object. Doesn't that break the rule that trash should > not be exposed to Python code while the collection is taking place? That's why I'm not touching anything here again for 2.4 <wink>. For builtin weakrefs to trash, it doesn't matter -- they get cleared before gc invokes any callbacks, so the worst a callback on a builtin-weakref-to-trash-object could do with its argument is resurrect a weakref than returns None if you call it. BFD. But I think we could concoct tests where instances of a weakref subclass did arbitrary damage in callbacks, via attributes bound to trash objects. The current "invoke the callback iff the weakref is reachable" avoids getting in trouble there, and I had weakref subclasses in the back of my mind when "settling" for that. We should write some nasty tests of weakref subclasses with callbacks interacting with gc -- there aren't any now. From tim.peters at gmail.com Thu Nov 4 21:11:32 2004 From: tim.peters at gmail.com (Tim Peters) Date: Thu Nov 4 21:11:38 2004 Subject: [Python-Dev] weakref gc semantics In-Reply-To: <8AFE1902-2E91-11D9-AF2F-000A95A50FB2@fuhm.net> References: <1f7befae04110220402077a6fa@mail.gmail.com> <8AFE1902-2E91-11D9-AF2F-000A95A50FB2@fuhm.net> Message-ID: <1f7befae041104121160f2b4ec@mail.gmail.com> [James Y Knight] > An unreachable callback function can also have visible effects. Quite true -- it might even reformat your hard drives <wink>. > ... > From what I'm hearing here, the following two pieces of code would then > have different behavior under patch-callback: > > """ > def C_gone(ignored): > print "An object of type C went away." > > class C: > def __init__(self): > self.wr = weakref.ref(self, C_gone) > """ > > and > > """ > class C: > def __init__(self): > def C_gone(ignored): > print "An object of type C went away." > self.wr = weakref.ref(self, C_gone) > """ > > If that's correct, it sounds like a bad idea to me. Assuming you're assuming the same driver is getting used after these snippets, yes, under patch-callback an instance of C in cyclic trash in the first case would trigger the callback, but not in the second case. I don't know why you believe that's bad: perhaps you could articulate a precise set of rules for what would be "good"? Maybe that's even the set of rules Python 2.4b2 actually uses. They seem more or less arbitrary to me (and so do patch-callback's -- and so do patch-finalizer's). For example, you can fiddle your examples above to get the same kind of "hmm, in one case it does, in the other case it doesn't" outcome in 2.4b2 by manipulating whether the weakref is externally reachable, instead of manipulating the reachability of the callback. Is that change in *what* you fiddle essentially different, so that it's "not bad" to get different outcomes when weakref reachability is fiddled? I don't think it's obvious, but under patch-finalizer they both get the same outcome (the callback is invoked, and the weakref gets deallocated before gc.garbage gets built). OTOH, if you make C_gone a method of C, and use self.C_gone as the callback, then the outcome is different: then the callback doesn't get called, and c is actually resurrected under patch-finalizer, even in the driver's first print "creating non-cyclic C instance" c = C() case (although the print stmt is no longer truthful then). From barry at python.org Thu Nov 4 22:06:08 2004 From: barry at python.org (Barry Warsaw) Date: Thu Nov 4 22:06:18 2004 Subject: [Python-Dev] Re: interlocking dependencies on the path to a release In-Reply-To: <41896727.7030206@v.loewis.de> References: <4188BE21.6040205@interlink.com.au> <41896727.7030206@v.loewis.de> Message-ID: <1099602368.17767.77.camel@geddy.wooz.org> On Wed, 2004-11-03 at 18:17, "Martin v. L?wis" wrote: > I think aiming at rewriting ht2html > in Python for 2.5 would be a worthwhile goal (I have dropped the > idea that conversion to xml would be worthwhile). Did you mean the Doc/perl stuff instead of ht2html? The latter is already Python (but you knew that). > Then, the release manager could trivially build the documentation. > For that to work, it is necessary that the documentation builds out > of the box; for that to happen, it is necessary that a broad audience > can build the documentation, so that problems are detected long > before the release (i.e. short after a checkin - or better yet, > before). For the PostScript/PDF release, one certainly needs TeX, > but that is free software. Actually, building the html documentation isn't hard, and I generally do it whenever I'm making a change to .tex files. I may not always write Fred-happy markup but I try to make sure it's at least syntactically correct. It's also a useful thing to have around for when there's no net access. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 307 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-dev/attachments/20041104/cdd83d0e/attachment.pgp From greg at cosc.canterbury.ac.nz Fri Nov 5 01:45:15 2004 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Fri Nov 5 01:45:22 2004 Subject: [Python-Dev] Re: weakref gc semantics In-Reply-To: <20041104173835.GA15004@mems-exchange.org> References: <1f7befae04110220402077a6fa@mail.gmail.com> <1f7befae041103212576c59c42@mail.gmail.com> <20041104173835.GA15004@mems-exchange.org> Message-ID: <418ACD1B.9060000@cosc.canterbury.ac.nz> Neil Schemenauer wrote: > I think there may be one small problem though. Callbacks get passed > the weakref object. Doesn't that break the rule that trash should > not be exposed to Python code while the collection is taking place? Maybe the callback shouldn't be passed the weakref object? I don't see what good that does anyway, since the weakref is just an empty shell by then, isn't it? Greg From tim.peters at gmail.com Fri Nov 5 02:42:15 2004 From: tim.peters at gmail.com (Tim Peters) Date: Fri Nov 5 02:42:21 2004 Subject: [Python-Dev] Re: weakref gc semantics In-Reply-To: <418ACD1B.9060000@cosc.canterbury.ac.nz> References: <1f7befae04110220402077a6fa@mail.gmail.com> <1f7befae041103212576c59c42@mail.gmail.com> <20041104173835.GA15004@mems-exchange.org> <418ACD1B.9060000@cosc.canterbury.ac.nz> Message-ID: <1f7befae04110417421a072d75@mail.gmail.com> [Greg Ewing] > Maybe the callback shouldn't be passed the weakref object? > I don't see what good that does anyway, since the weakref > is just an empty shell by then, isn't it? If it's a builtin weakref object, yes. Weakref subclasses can add useful data, though. The way Python's weak dictionaries exploit this was explained earlier in the thread. From anthony at interlink.com.au Fri Nov 5 03:53:54 2004 From: anthony at interlink.com.au (Anthony Baxter) Date: Fri Nov 5 03:54:20 2004 Subject: [Python-Dev] Policy about old Python versions In-Reply-To: <20041104141632.GA27459@panix.com> References: <A128D751272CD411BC9200508BC2194D053C7B5D@escpl.tcp.com.ar> <20041104141632.GA27459@panix.com> Message-ID: <418AEB42.8040306@interlink.com.au> Aahz wrote: > As soon as the new version is released, if there is no bugfix planned > for that issue. Two versions later (i.e. 2.3 in this case) even if it > looks like it ought to be fixed in 2.1 -- we haven't been doing bugfixes > more than one version old. That's not to say that if someone wants to cut a 2.2.4 or 2.1.4 I'm going to stop them. I've no interest in maintaining more than trunk plus the most recent maint branch. As for the original query, about what to do with bugs filed against 2.1.x - I'd suggest closing them. There's a vanishingly small chance that there will ever be another 2.1 or 2.2 release. Anthony From bac at OCF.Berkeley.EDU Fri Nov 5 06:37:53 2004 From: bac at OCF.Berkeley.EDU (Brett C.) Date: Fri Nov 5 06:38:14 2004 Subject: [Python-Dev] TRUNK FROZEN for 2.4b2 release from 0000UTC 3rdNovember (tomorrrow) In-Reply-To: <000001c4c22c$1daf6500$e841fea9@oemcomputer> References: <000001c4c22c$1daf6500$e841fea9@oemcomputer> Message-ID: <418B11B1.8000106@ocf.berkeley.edu> Raymond Hettinger wrote: > [Bob Ippolito] > >>The thing is that the incompatible change is going in, but the >>compatible changes (the reason for the incompatible change) are not. >>This is very peculiar. > > > [Anthony] > >>Bear in mind that we've had _no_ macpython releases in the 2.4 cycle >>so far, that I'm aware of. This is really a platform-specific build >>issue. > > > > Poor Bob, this reasonable request has been kicked around too much. > > It is a simple thing that strips unnecessary stuff out of the build. > Will someone who has a Mac please apply. > If someone else wants to apply it I have uploaded my revision of Bob's patch to SF (just make sure to run autoconf; patch is at http://www.python.org/sf/1035255 ). And I don't see why a Mac user needs to apply it since I have personally tested it to the best of my abilities. But as I have said I will not apply unless Anthony gives me a clear "yes" to do so. Call me over-cautious and annoying. -Brett From mwh at python.net Fri Nov 5 12:48:16 2004 From: mwh at python.net (Michael Hudson) Date: Fri Nov 5 12:48:18 2004 Subject: [Python-Dev] Re: weakref gc semantics In-Reply-To: <1f7befae04110411327af9bf3d@mail.gmail.com> (Tim Peters's message of "Thu, 4 Nov 2004 14:32:51 -0500") References: <1f7befae04110220402077a6fa@mail.gmail.com> <1f7befae041103212576c59c42@mail.gmail.com> <20041104173835.GA15004@mems-exchange.org> <1f7befae04110411327af9bf3d@mail.gmail.com> Message-ID: <2m7jp05xfj.fsf@starship.python.net> Tim Peters <tim.peters@gmail.com> writes: [stuff i admittedly haven't thought terribly hard about] > The former is accurate independent of whether we're talking about > cyclic trash now, but that latter would not be. For example, > > def callback(ignore): > print 'hi' > > class C: > pass > > c = C() > w = weakref.ref(c, callback) > w = None > c = None > > The callback outlives c, but the callback doesn't get invoked, because > w didn't outlive c. The callback does get invoked if the order of the > last two lines is swapped. It's easy to explain this in terms of > which of {c, w} goes away first, but not in terms of callback's > lifetime. As disclaimed above, I haven't thought about this a great deal, but I would expect it to be the lifetime of the weakref which matters -- in the not-all-that-unlikley case of the callback not being a lambda or local function, it is *always* going to be reachable, and indeed is never going to die until the process exits... I take it the changes you're proposing aren't going to change the semantics of the quoted code? Cheers, mwh -- I'm about to search Google for contract assassins to go to Iomega and HP's programming groups and kill everyone there with some kind of electrically charged rusty barbed thing. -- http://bofhcam.org/journal/journal.html, 2002-01-08 From mwh at python.net Fri Nov 5 12:50:51 2004 From: mwh at python.net (Michael Hudson) Date: Fri Nov 5 12:50:55 2004 Subject: [Python-Dev] Re: interlocking dependencies on the path to a release In-Reply-To: <41896727.7030206@v.loewis.de> ( =?iso-8859-1?q?Martin_v._L=F6wis's_message_of?= "Thu, 04 Nov 2004 00:17:59 +0100") References: <4188BE21.6040205@interlink.com.au> <41896727.7030206@v.loewis.de> Message-ID: <2m3bzo5xb8.fsf@starship.python.net> "Martin v. L?wis" <martin@v.loewis.de> writes: > Also, I think it would be good to eliminate Fred's role altogether: > the documentation build process should be much more automatic, and > less relying on third-party tools. I think aiming at rewriting ht2html > in Python for 2.5 would be a worthwhile goal (I have dropped the > idea that conversion to xml would be worthwhile). Err, do you mean latex2html? If so, I'm not at all sure that's realistic. Cheers, mwh -- After a heavy night I travelled on, my face toward home - the comma being by no means guaranteed. -- paraphrased from cam.misc From jim at zope.com Fri Nov 5 15:25:20 2004 From: jim at zope.com (Jim Fulton) Date: Fri Nov 5 15:25:31 2004 Subject: [Python-Dev] Re: weakref gc semantics In-Reply-To: <418ACD1B.9060000@cosc.canterbury.ac.nz> References: <1f7befae04110220402077a6fa@mail.gmail.com> <1f7befae041103212576c59c42@mail.gmail.com> <20041104173835.GA15004@mems-exchange.org> <418ACD1B.9060000@cosc.canterbury.ac.nz> Message-ID: <418B8D50.4090908@zope.com> Greg Ewing wrote: > Neil Schemenauer wrote: > >> I think there may be one small problem though. Callbacks get passed >> the weakref object. Doesn't that break the rule that trash should >> not be exposed to Python code while the collection is taking place? > > > Maybe the callback shouldn't be passed the weakref object? > I don't see what good that does anyway, since the weakref > is just an empty shell by then, isn't it? No. It can still be used as a key. It still has it's original hash value and will compare equal to other weakrefs to the same original object. Weak-key dictionaries depend on this. In addition, as Tim points out, it could be a weakref-subclass instance and could have additional useful data. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From tim.peters at gmail.com Fri Nov 5 16:05:23 2004 From: tim.peters at gmail.com (Tim Peters) Date: Fri Nov 5 16:05:26 2004 Subject: [Python-Dev] Re: weakref gc semantics In-Reply-To: <418B8D50.4090908@zope.com> References: <1f7befae04110220402077a6fa@mail.gmail.com> <1f7befae041103212576c59c42@mail.gmail.com> <20041104173835.GA15004@mems-exchange.org> <418ACD1B.9060000@cosc.canterbury.ac.nz> <418B8D50.4090908@zope.com> Message-ID: <1f7befae041105070549529618@mail.gmail.com> [Greg Ewing] >> Maybe the callback shouldn't be passed the weakref object? >> I don't see what good that does anyway, since the weakref >> is just an empty shell by then, isn't it? [Jim Fulton] > No. It can still be used as a key. It still has it's original > hash value and will compare equal to other weakrefs to the same > original object. Weak-key dictionaries depend on this. Right. > In addition, as Tim points out, it could be a weakref-subclass instance > and could have additional useful data. Adn weak-value dictionaries depend on that; from weakref.py: class KeyedRef(ref): """Specialized reference that includes a key corresponding to the value. This is used in the WeakValueDictionary to avoid having to create a function object for each key stored in the mapping. A shared callback object can use the 'key' attribute of a KeyedRef instead of getting a reference to the key from an enclosing scope. """ ... From pje at telecommunity.com Fri Nov 5 16:24:24 2004 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri Nov 5 16:23:34 2004 Subject: [Python-Dev] Re: interlocking dependencies on the path to a release In-Reply-To: <2m3bzo5xb8.fsf@starship.python.net> References: <41896727.7030206@v.loewis.de> <4188BE21.6040205@interlink.com.au> <41896727.7030206@v.loewis.de> Message-ID: <5.1.1.6.0.20041105101943.02b94c80@mail.telecommunity.com> At 11:50 AM 11/5/04 +0000, Michael Hudson wrote: >"Martin v. L?wis" <martin@v.loewis.de> writes: > > > Also, I think it would be good to eliminate Fred's role altogether: > > the documentation build process should be much more automatic, and > > less relying on third-party tools. I think aiming at rewriting ht2html > > in Python for 2.5 would be a worthwhile goal (I have dropped the > > idea that conversion to xml would be worthwhile). > >Err, do you mean latex2html? If so, I'm not at all sure that's >realistic. I've dabbled in the guts of latex2html before; it's certainly not pretty. IMO a better long term option might be to use the Python docutils and migrate to reStructuredText, since there are a bevy of backends available for latex, PDF, HTML, etc. They would probably need more work before they'll be suitable to handle the entire doc generation process, though. From tim.peters at gmail.com Fri Nov 5 16:45:42 2004 From: tim.peters at gmail.com (Tim Peters) Date: Fri Nov 5 16:45:47 2004 Subject: [Python-Dev] Re: weakref gc semantics In-Reply-To: <2m7jp05xfj.fsf@starship.python.net> References: <1f7befae04110220402077a6fa@mail.gmail.com> <1f7befae041103212576c59c42@mail.gmail.com> <20041104173835.GA15004@mems-exchange.org> <1f7befae04110411327af9bf3d@mail.gmail.com> <2m7jp05xfj.fsf@starship.python.net> Message-ID: <1f7befae04110507454a504cf6@mail.gmail.com> [Michael Hudson] > As disclaimed above, I haven't thought about this a great deal, but I > would expect it to be the lifetime of the weakref which matters -- in > the not-all-that-unlikley case of the callback not being a lambda or > local function, it is *always* going to be reachable, and indeed is > never going to die until the process exits... A difficulty remaining is that when a weakref and its referent both become unreachable simultaneously, relative lifetime becomes fuzzy. Even when, in the "simple" examples, I write weakref = None referent = None # callback doesn't trigger or referent = None # callback does trigger weakref = None the idea that relative lifetime controls the outcome in a *predictable* way depends on that CPython uses refcounting as its primary gc strategy. Indeed, Python the language (as opposed to CPython the implementation) really can't guarantee that the callback does or doesn't trigger in either simple example above. These fuzzy issues have become important because the lack of clarity about exact intended semantics has left us with segfault-producing bugs in the interaction between cyclic gc and weakrefs, and we've been thru more than one round of those now. A complication is that, perhaps contrary to initial expectation, outside of contrived tests I'm not sure callbacks that *aren't* "lambdas or local functions" actually exist. In the core, the two flavors of weak dict use very different strategies, but both end up using a unique function per dict instance as the callback, created in the weak dict's __init__ and bound to an attribute of the dict. So these are guaranteed to create "fuzzy situations" when a weak dict becomes part of cyclic trash (the dict, and the weakrefs it holds, and their callbacks, generally all become unreachable at the same instant). > I take it the changes you're proposing aren't going to change the > semantics of the quoted code? I'm not proposing anything specific, and if it comes to it I bet I could live with 2.4b2 forever. Its rules seem ultimately arbitrary, though, so aren't really attractive. Other rules are possible, and should be considered. For the first time, I think we finally have a deep understanding of what we can and cannot *get away* with now wrt preventing segfault bugs (and other related nastiness). But that understanding doesn't reduce the universe of acceptable (bug-free and defensible) policies to what 2.4b2 implements. In any case, no change to gcmodule.c's weakref policy would change anything about how CPython behaves in cases where refcounting does the whole job. So the answer to your literal <wink> question is "no". The behaviors of > def callback(ignore): > print 'hi' > > class C: > pass > > c = C() > w = weakref.ref(c, callback) > w = None > c = None kinds of examples are wholly determined by refcount lifetime semantics in CPython, and that won't change. From trentm at ActiveState.com Fri Nov 5 19:29:40 2004 From: trentm at ActiveState.com (Trent Mick) Date: Fri Nov 5 19:31:04 2004 Subject: [Python-Dev] Re: interlocking dependencies on the path to a release In-Reply-To: <5.1.1.6.0.20041105101943.02b94c80@mail.telecommunity.com> References: <41896727.7030206@v.loewis.de> <4188BE21.6040205@interlink.com.au> <41896727.7030206@v.loewis.de> <5.1.1.6.0.20041105101943.02b94c80@mail.telecommunity.com> Message-ID: <418BC694.9010306@activestate.com> Phillip J. Eby wrote: > I've dabbled in the guts of latex2html before; it's certainly not pretty. > > IMO a better long term option might be to use the Python docutils and > migrate to reStructuredText, since there are a bevy of backends > available for latex, PDF, HTML, etc. They would probably need more work > before they'll be suitable to handle the entire doc generation process, > though. What do people think about docbook? I'm impressed with the kind of doc output that, for example, the PHP guys and Mark Pilgrim's Dive Into Python and the Subversion Book, can generate from docbook sources. I'm not suggesting any whole hog re-write of the Python docs anytime soon, just wanted to feel it out. Trent -- Trent Mick trentm@activestate.com From trentm at ActiveState.com Fri Nov 5 19:26:02 2004 From: trentm at ActiveState.com (Trent Mick) Date: Fri Nov 5 19:42:48 2004 Subject: [Python-Dev] Policy about old Python versions In-Reply-To: <418AEB42.8040306@interlink.com.au> References: <A128D751272CD411BC9200508BC2194D053C7B5D@escpl.tcp.com.ar> <20041104141632.GA27459@panix.com> <418AEB42.8040306@interlink.com.au> Message-ID: <418BC5BA.7090108@activestate.com> Anthony Baxter wrote: > As for the original query, about what to do with bugs filed against > 2.1.x - I'd suggest closing them. There's a vanishingly small chance > that there will ever be another 2.1 or 2.2 release. Maybe a "Known Issues in Python 2.1" PEP which refers to these bug numbers and then close the bugs? I don't want to make more work for people, though. Trent -- Trent Mick trentm@activestate.com From bac at OCF.Berkeley.EDU Fri Nov 5 20:42:43 2004 From: bac at OCF.Berkeley.EDU (Brett C.) Date: Fri Nov 5 20:43:06 2004 Subject: [Python-Dev] Re: interlocking dependencies on the path to a release In-Reply-To: <5.1.1.6.0.20041105101943.02b94c80@mail.telecommunity.com> References: <41896727.7030206@v.loewis.de> <4188BE21.6040205@interlink.com.au> <41896727.7030206@v.loewis.de> <5.1.1.6.0.20041105101943.02b94c80@mail.telecommunity.com> Message-ID: <418BD7B3.7050306@ocf.berkeley.edu> Phillip J. Eby wrote: > At 11:50 AM 11/5/04 +0000, Michael Hudson wrote: > >> "Martin v. L?wis" <martin@v.loewis.de> writes: >> >> > Also, I think it would be good to eliminate Fred's role altogether: >> > the documentation build process should be much more automatic, and >> > less relying on third-party tools. I think aiming at rewriting ht2html >> > in Python for 2.5 would be a worthwhile goal (I have dropped the >> > idea that conversion to xml would be worthwhile). >> >> Err, do you mean latex2html? If so, I'm not at all sure that's >> realistic. > > > I've dabbled in the guts of latex2html before; it's certainly not pretty. > > IMO a better long term option might be to use the Python docutils and > migrate to reStructuredText, since there are a bevy of backends > available for latex, PDF, HTML, etc. They would probably need more work > before they'll be suitable to handle the entire doc generation process, > though. > This has been proposed before but has been shot down. If I remember correctly the reasoning was that LaTeX gave us more control and thus served the purpose better for the official docs. -Brett From pje at telecommunity.com Fri Nov 5 21:04:21 2004 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri Nov 5 21:03:32 2004 Subject: [Python-Dev] Re: interlocking dependencies on the path to a release In-Reply-To: <418BD7B3.7050306@ocf.berkeley.edu> References: <5.1.1.6.0.20041105101943.02b94c80@mail.telecommunity.com> <41896727.7030206@v.loewis.de> <4188BE21.6040205@interlink.com.au> <41896727.7030206@v.loewis.de> <5.1.1.6.0.20041105101943.02b94c80@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20041105145001.02dee410@mail.telecommunity.com> At 11:42 AM 11/5/04 -0800, Brett C. wrote: >Phillip J. Eby wrote: >>At 11:50 AM 11/5/04 +0000, Michael Hudson wrote: >>>Err, do you mean latex2html? If so, I'm not at all sure that's >>>realistic. >> >>I've dabbled in the guts of latex2html before; it's certainly not pretty. >>IMO a better long term option might be to use the Python docutils and >>migrate to reStructuredText, since there are a bevy of backends available >>for latex, PDF, HTML, etc. They would probably need more work before >>they'll be suitable to handle the entire doc generation process, though. > >This has been proposed before but has been shot down. If I remember >correctly the reasoning was that LaTeX gave us more control and thus >served the purpose better for the official docs. More control of what? I thought that reST was specifically designed to accommodate all of the Python-specific markup we're using in the latex docs. As a matter of language expressiveness, as far as I can tell, reST accomodates marking up both short strings and long blocks, with application-specific markup, so I don't really understand why there shouldn't be a largely 1:1 mapping between the markup systems. To be fair, however, here are the downsides I'm aware of: * There are more tools that support editing LaTeX than reST * Python code would have to be written to replace the current Python-specific LaTeX macros with reST "interpreted text modes" and "block directives" * Conversion could be very time consuming, for the parts that can't be automated * There is a risk that the needed back-ends may not be sufficiently mature or feature-rich. For example, I don't think there's an HTML back-end that can generate pages for different nodes, with navigation links. I don't know if there's a GNU 'info' backend, either. Anyway, I thought the whole point behind the docutils initiative was to create a doc format that could be used anywhere Python needs docs, ala Perl's POD. If there are problems with reST, maybe we should let the docutils folks know what additional requirements exist, so that it can (eventually) be used for this purpose. From tim.peters at gmail.com Fri Nov 5 21:07:46 2004 From: tim.peters at gmail.com (Tim Peters) Date: Fri Nov 5 22:54:39 2004 Subject: [Python-Dev] patch-finalizer vs Zope3 In-Reply-To: <20041104171957.A0C633B8038@smtp.zope.com> References: <20041104171957.A0C633B8038@smtp.zope.com> Message-ID: <1f7befae041105120727ae399e@mail.gmail.com> [Tim Peters] > FYI, and without taking time for non-trivial analysis. I ran the non-ZEO > Zope3 tests with a Python patched to treat weakrefs with callbacks as > reachable (this was patch-finalizer.txt, attached to a msg I sent to > python-dev last night). > > The first test to leave stuff in gc.garbage then: > > The following test left garbage: > test_created (bugtracker.browser.tests.test_bug.BugBaseViewTest) > 65 > > where 65 is len(gc.garbage) after the test finished. OK, that actually wasn't the first test to produce garbage, it was just the first time test.py *noticed* gc.garbage wasn't empty. The first test to produce garbage was BuddyCityState, and it produces garbage in isolation: C:\Code\Zope3>\code\python\PCbuild\python_d.exe test.py -vv . BuddyCityState Configuration file found. Running UNIT tests at level 1 Running UNIT tests from C:\Code\Zope3 BuddyCityState (buddydemo.buddy) ... ok The following test left garbage: BuddyCityState (buddydemo.buddy) 28 entries in gc.garbage: 0: <weakref at 03C86F20; to 'InterfaceClass' at 012B5930 (IWriteContainer)> 0x12B5930 is <InterfaceClass zope.app.container.interfaces.IWriteContainer> 1: <weakref at 03C8E080; to 'InterfaceClass' at 00B5EAB8 (Interface)> 0xB5EAB8 is <InterfaceClass zope.interface.Interface> 2: <weakref at 03C8E1A0; to 'Surrogate' at 03C8D4D0> 0x3C8D4D0 is <Surrogate(<InterfaceClass zope.app.container.interfaces.IWriteContainer>)> 3: <weakref at 03C8E278; to 'InterfaceClass' at 012B5578 (IObjectEvent)> 0x12B5578 is <InterfaceClass zope.app.event.interfaces.IObjectEvent> 4: <weakref at 03C8E398; to 'Surrogate' at 03C8D6C8> 0x3C8D6C8 is <Surrogate(<InterfaceClass zope.app.event.interfaces.IObjectEvent>)> 5: <weakref at 03C8E3E0; to 'InterfaceClass' at 01197230 (IHTTPRequest)> 0x1197230 is <InterfaceClass zope.publisher.interfaces.http.IHTTPRequest> 6: <weakref at 03C8E500; to 'InterfaceClass' at 0118ACE8 (IRequest)> 0x118ACE8 is <InterfaceClass zope.publisher.interfaces.IRequest> 7: <weakref at 03C8E620; to 'InterfaceClass' at 0118A818 (IPublisherRequest)> 0x118A818 is <InterfaceClass zope.publisher.interfaces.IPublisherRequest> 8: <weakref at 03C8E740; to 'InterfaceClass' at 0118A5E8 (IPublicationRequest)> 0x118A5E8 is <InterfaceClass zope.publisher.interfaces.IPublicationRequest> 9: <weakref at 03C8E860; to 'InterfaceClass' at 00DB9690 (IPresentationRequest)> 0xDB9690 is <InterfaceClass zope.component.interfaces.IPresentationRequest> 10: <weakref at 03C8E980; to 'Surrogate' at 03C8DAB8> 0x3C8DAB8 is <Surrogate(<InterfaceClass zope.component.interfaces.IPresentationRequest>)> 11: <weakref at 03C8E9C8; to 'InterfaceClass' at 00DA0188 (IParticipation)> 0xDA0188 is <InterfaceClass zope.security.interfaces.IParticipation> 12: <weakref at 03C8EAE8; to 'Surrogate' at 03C8DB28> 0x3C8DB28 is <Surrogate(<InterfaceClass zope.security.interfaces.IParticipation>)> 13: <weakref at 03C8EB30; to 'Surrogate' at 03C8DA10> 0x3C8DA10 is <Surrogate(<InterfaceClass zope.publisher.interfaces.IPublicationRequest>)> 14: <weakref at 03C8EB78; to 'Surrogate' at 03C8DA10> 0x3C8DA10 is <Surrogate(<InterfaceClass zope.publisher.interfaces.IPublicationRequest>)> 15: <weakref at 03C8EBC0; to 'Surrogate' at 03C8D968> 0x3C8D968 is <Surrogate(<InterfaceClass zope.publisher.interfaces.IPublisherRequest>)> 16: <weakref at 03C8EC08; to 'InterfaceClass' at 0118AC08 (IApplicationRequest)> 0x118AC08 is <InterfaceClass zope.publisher.interfaces.IApplicationRequest> 17: <weakref at 03C8ED28; to 'InterfaceClass' at 01177F18 (IEnumerableMapping)> 0x1177F18 is <InterfaceClass zope.interface.common.mapping.IEnumerableMapping> 18: <weakref at 03C8EE48; to 'InterfaceClass' at 01177D58 (IReadMapping)> 0x1177D58 is <InterfaceClass zope.interface.common.mapping.IReadMapping> 19: <weakref at 03C8EF68; to 'InterfaceClass' at 01177C40 (IItemMapping)> 0x1177C40 is <InterfaceClass zope.interface.common.mapping.IItemMapping> 20: <weakref at 03C8F0C8; to 'Surrogate' at 03C8DD20> 0x3C8DD20 is <Surrogate(<InterfaceClass zope.interface.common.mapping.IItemMapping>)> 21: <weakref at 03C8F110; to 'Surrogate' at 03C8DC78> 0x3C8DC78 is <Surrogate(<InterfaceClass zope.interface.common.mapping.IReadMapping>)> 22: <weakref at 03C8F158; to 'Surrogate' at 03C8DBD0> 0x3C8DBD0 is <Surrogate(<InterfaceClass zope.interface.common.mapping.IEnumerableMapping>)> 23: <weakref at 03C8F1A0; to 'Surrogate' at 03C8D9D8> 0x3C8D9D8 is <Surrogate(<InterfaceClass zope.publisher.interfaces.IApplicationRequest>)> 24: <weakref at 03C8F1E8; to 'Surrogate' at 03C8D8C0> 0x3C8D8C0 is <Surrogate(<InterfaceClass zope.publisher.interfaces.IRequest>)> 25: <weakref at 03C8F230; to 'Surrogate' at 03C8D8C0> 0x3C8D8C0 is <Surrogate(<InterfaceClass zope.publisher.interfaces.IRequest>)> 26: <weakref at 03C8F278; to 'Surrogate' at 03C8D8C0> 0x3C8D8C0 is <Surrogate(<InterfaceClass zope.publisher.interfaces.IRequest>)> 27: <weakref at 03C8F2C0; to 'Surrogate' at 03C8D818> 0x3C8D818 is <Surrogate(<InterfaceClass zope.publisher.interfaces.http.IHTTPRequest>)> [703156 refs] C:\Code\Zope3> It appears generally true that, like in that test, gc.garbage gets filled with (hundreds of thousands of) weakrefs specifically due to adaptation machinery. By inspection, there are a lot of weakrefs flying around there <wink>. From tim.peters at gmail.com Fri Nov 5 23:06:41 2004 From: tim.peters at gmail.com (Tim Peters) Date: Fri Nov 5 23:06:46 2004 Subject: [Python-Dev] Code coverage tool updated In-Reply-To: <418A269B.6010207@livinglogic.de> References: <41864572.4000806@livinglogic.de> <16774.45652.835103.327656@montanaro.dyndns.org> <4187765D.3000705@livinglogic.de> <4187CA1C.9050508@livinglogic.de> <1f7befae04110210537c35e514@mail.gmail.com> <418A269B.6010207@livinglogic.de> Message-ID: <1f7befae0411051406711618cf@mail.gmail.com> [Walter D?rwald] > I guess it's not worth it to try to fix doctest/trace to > provide sourcecode for doctest code, but IMHO trace should > be able to survive a doctest. Sure! The assert also triggers for "@test" on my box, BTW -- and also did in 2.3.4. > Removing the assert statement, so that trace.py runs to > completion shows a different problem: There are only > 32 files covered according to the trace output. The > complete test log can be found here: > http://styx.livinglogic.de/~walter/brokentrace.txt > > The trace call looked like this: > ./python ../../../trace.py --count --summary --missing Lib/test/regrtest.py > with ../../../trace.py being the trace.py from current > CVS with the assert statement removed. > > So am I doing something wrong or is trace.py broken? Sorry, I don't know. trace.py is like voting to me -- I'm highly in favor of it, but never have time for it <0.5 wink>. I only dropped in here to explain the source of the synthesized doctest "file name". FWIW, doing what you did with current CVS Python on Windows, I get results similar to yours: only 30-some modules reported at the end. Under my 2.3.4 installation instead, the same thing reports on over 300 modules, so best guess is that trace.py is indeed suffering breakage introduced in 2.4. No idea about specifics, though. From kbk at shore.net Sat Nov 6 04:08:08 2004 From: kbk at shore.net (Kurt B. Kaiser) Date: Sat Nov 6 04:08:15 2004 Subject: [Python-Dev] Weekly Python Patch/Bug Summary Message-ID: <200411060308.iA6388E3022594@h006008a7bda6.ne.client2.attbi.com> Patch / Bug Summary ___________________ Patches : 241 open ( -2) / 2683 closed ( +5) / 2924 total ( +3) Bugs : 784 open ( -5) / 4567 closed (+22) / 5351 total (+17) RFE : 158 open ( +2) / 131 closed ( +0) / 289 total ( +2) New / Reopened Patches ______________________ New BaseSMTPServer module (2004-10-30) http://python.org/sf/1057417 opened by Mark D. Roth chr, ord, unichr documentation updates (2004-10-31) http://python.org/sf/1057588 opened by Mike Brown bdist_rpm not able to compile multiple rpm packages (2004-11-04) http://python.org/sf/1060577 opened by Mihai Ibanescu Patches Closed ______________ No ValueError for safe_substitute() (2004-10-29) http://python.org/sf/1056967 closed by bwarsaw typo in comment (unicodeobject.h) (2004-10-28) http://python.org/sf/1056231 closed by rhettinger ref. manual talks of 'sequence' instead of 'iterable' (2003-10-23) http://python.org/sf/829073 closed by fdrake Fix for Unix macro expansion in citetitle (2004-10-26) http://python.org/sf/1054715 closed by fdrake Remove CoreServices / CoreFoundation dependencies in core (2004-09-26) http://python.org/sf/1035255 closed by rhettinger New / Reopened Bugs ___________________ add link in time module to calendar.timegm() (2004-10-30) CLOSED http://python.org/sf/1057535 opened by Jeff Shute compiler.transformer, "from module import *" (2004-10-31) http://python.org/sf/1057835 opened by Felix Wiemann os.utime() doesn't work on WinME (2004-11-01) CLOSED http://python.org/sf/1057992 opened by Raymond Hettinger test_traceback.py fails on WinME (2004-11-01) CLOSED http://python.org/sf/1057993 opened by Raymond Hettinger Can't read some http URLs using neither urllib, nor urllib2 (2004-11-01) http://python.org/sf/1058059 opened by Vsevolod Novikov Blatantly false statement in the Unicode section (2004-11-01) CLOSED http://python.org/sf/1058351 opened by Olivier Verdier r'\10' as replacement pattern loops in compilation (2004-11-02) http://python.org/sf/1058786 opened by Nick Maclaren Legacy requirements: Incorrect comments? (2004-11-02) http://python.org/sf/1058937 opened by Geoffrey T. Dairiki install_lib fails under Python 2.1 (2004-11-02) http://python.org/sf/1058960 opened by Geoffrey T. Dairiki distutil bdist hardcodes the python location (2004-11-02) http://python.org/sf/1059244 opened by Steve Menard errors=ignore broken on StreamWriter? (2004-11-03) CLOSED http://python.org/sf/1059748 opened by Wummel incorrect "'yield' outside function" error (2004-11-04) http://python.org/sf/1060135 opened by Willem Broekema os.utime does not work on directories under Windows (2004-11-04) CLOSED http://python.org/sf/1060212 opened by Boyle Buffer overflow in socketmodule.c (2004-11-04) http://python.org/sf/1060396 opened by Mihai Ibanescu Remove .min() and .max() from Decimal (2004-11-04) http://python.org/sf/1060644 reopened by facundobatista Remove .min() and .max() from Decimal (2004-11-04) http://python.org/sf/1060644 opened by Facundo Batista Error in difflib docs (2004-11-05) CLOSED http://python.org/sf/1060825 opened by Kent Johnson email.Parser does not recognize attachment (2004-11-05) CLOSED http://python.org/sf/1060941 opened by Martin Muenstermann Bugs Closed ___________ weakref callback vs gc vs threads (2004-10-27) http://python.org/sf/1055820 closed by tim_one add link in time module to calendar.timegm() (2004-10-31) http://python.org/sf/1057535 closed by jlgijsbers os.rmtree error handling was always broken (2004-10-18) http://python.org/sf/1048941 closed by jlgijsbers minor error in os.access (2004-10-29) http://python.org/sf/1056498 closed by jlgijsbers Attempt to run all atexit handlers (2004-10-22) http://python.org/sf/1052242 closed by montanaro email.Utils not mentioned (2004-09-17) http://python.org/sf/1030118 closed by bwarsaw os.utime() doesn't work on WinME (2004-11-01) http://python.org/sf/1057992 closed by rhettinger test_traceback.py fails on WinME (2004-11-01) http://python.org/sf/1057993 closed by perky email.Message does not allow iteration (2004-08-26) http://python.org/sf/1017329 closed by bwarsaw Blatantly false statement in the Unicode section (2004-11-02) http://python.org/sf/1058351 closed by perky File write examples are inadequate (2002-10-09) http://python.org/sf/621057 closed by fdrake Clarify trailing comma in func arg list (2003-09-01) http://python.org/sf/798652 closed by fdrake Install-time module compilation fails with 2.4b1 (2004-10-18) http://python.org/sf/1049003 closed by loewis errors=ignore broken on StreamWriter? (2004-11-03) http://python.org/sf/1059748 closed by lemburg latex2html convert bug (2003-12-30) http://python.org/sf/867556 closed by quiver os.utime does not work on directories under Windows (2004-11-04) http://python.org/sf/1060212 closed by tim_one Remove .min() and .max() from Decimal (2004-11-04) http://python.org/sf/1060644 closed by rhettinger LaTeX not required (2004-05-05) http://python.org/sf/948517 closed by fdrake ambiguity in os.tmpfile (2004-10-28) http://python.org/sf/1056515 closed by fdrake Duplicated <link rel (2004-02-18) http://python.org/sf/899423 closed by fdrake Error in difflib docs (2004-11-05) http://python.org/sf/1060825 closed by rhettinger email.Parser does not recognize attachment (2004-11-05) http://python.org/sf/1060941 closed by bwarsaw New / Reopened RFE __________________ lock-free data structures? (2004-11-03) http://python.org/sf/1059545 opened by Markus Elfring From martin at v.loewis.de Sat Nov 6 11:10:45 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat Nov 6 11:10:35 2004 Subject: [Python-Dev] Re: interlocking dependencies on the path to a release In-Reply-To: <2m3bzo5xb8.fsf@starship.python.net> References: <4188BE21.6040205@interlink.com.au> <41896727.7030206@v.loewis.de> <2m3bzo5xb8.fsf@starship.python.net> Message-ID: <418CA325.50105@v.loewis.de> Michael Hudson wrote: > Err, do you mean latex2html? If so, I'm not at all sure that's > realistic. We'll see. What I consider the difficult part of such a project (parsing the TeX) is already implemented, so we only need the HTML generation now. That should be straight-forward, even if laborious. Regards, Martin From martin at v.loewis.de Sat Nov 6 11:12:55 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat Nov 6 11:12:44 2004 Subject: [Python-Dev] Re: interlocking dependencies on the path to a release In-Reply-To: <5.1.1.6.0.20041105101943.02b94c80@mail.telecommunity.com> References: <41896727.7030206@v.loewis.de> <4188BE21.6040205@interlink.com.au> <41896727.7030206@v.loewis.de> <5.1.1.6.0.20041105101943.02b94c80@mail.telecommunity.com> Message-ID: <418CA3A7.709@v.loewis.de> Phillip J. Eby wrote: > I've dabbled in the guts of latex2html before; it's certainly not pretty. I wouldn't look into the source of latex2html at all. Instead, I would rewrite it from scratch, worrying only that the output stays the same (or sufficiently similar). > IMO a better long term option might be to use the Python docutils and > migrate to reStructuredText, since there are a bevy of backends > available for latex, PDF, HTML, etc. They would probably need more work > before they'll be suitable to handle the entire doc generation process, > though. I'm not convinced this would be a good idea. Having TeX as the primary source allows to produce well-formatted printed documentation. Regards, Martin From martin at v.loewis.de Sat Nov 6 11:14:09 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat Nov 6 11:13:58 2004 Subject: [Python-Dev] Re: interlocking dependencies on the path to a release In-Reply-To: <418BC694.9010306@activestate.com> References: <41896727.7030206@v.loewis.de> <4188BE21.6040205@interlink.com.au> <41896727.7030206@v.loewis.de> <5.1.1.6.0.20041105101943.02b94c80@mail.telecommunity.com> <418BC694.9010306@activestate.com> Message-ID: <418CA3F1.4050702@v.loewis.de> Trent Mick wrote: > What do people think about docbook? It's been a plan for several years now, and I spend some time into advancing the existing tools. Now I'm not convinced anymore that this is desirable - people don't want to learn new markup languages every day. Regards, Martin From martin at v.loewis.de Sat Nov 6 11:18:49 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat Nov 6 11:18:39 2004 Subject: [Python-Dev] Re: interlocking dependencies on the path to a release In-Reply-To: <5.1.1.6.0.20041105145001.02dee410@mail.telecommunity.com> References: <5.1.1.6.0.20041105101943.02b94c80@mail.telecommunity.com> <41896727.7030206@v.loewis.de> <4188BE21.6040205@interlink.com.au> <41896727.7030206@v.loewis.de> <5.1.1.6.0.20041105101943.02b94c80@mail.telecommunity.com> <5.1.1.6.0.20041105145001.02dee410@mail.telecommunity.com> Message-ID: <418CA509.2050103@v.loewis.de> Phillip J. Eby wrote: > More control of what? I thought that reST was specifically designed to > accommodate all of the Python-specific markup we're using in the latex > docs. How do you create a module index and a "global" index in reST? How do you express document inclusion (in the spirit of \input)? > As a matter of language expressiveness, as far as I can tell, reST > accomodates marking up both short strings and long blocks, with > application-specific markup, so I don't really understand why there > shouldn't be a largely 1:1 mapping between the markup systems. It's not about source code display. It is about the other 200 typographic features that we use in the Python documentation. > * Conversion could be very time consuming, for the parts that can't be > automated If the conversion cannot be fully automatic, that would indicate that there will be a loss of information. Likely, this loss is unacceptable. > Anyway, I thought the whole point behind the docutils initiative was to > create a doc format that could be used anywhere Python needs docs, ala > Perl's POD. I'm still uncertain what the point behind the docutils initiative is. Regards, Martin From mwh at python.net Sat Nov 6 13:58:10 2004 From: mwh at python.net (Michael Hudson) Date: Sat Nov 6 13:58:12 2004 Subject: [Python-Dev] Re: interlocking dependencies on the path to a release In-Reply-To: <418BC694.9010306@activestate.com> (Trent Mick's message of "Fri, 05 Nov 2004 10:29:40 -0800") References: <41896727.7030206@v.loewis.de> <4188BE21.6040205@interlink.com.au> <41896727.7030206@v.loewis.de> <5.1.1.6.0.20041105101943.02b94c80@mail.telecommunity.com> <418BC694.9010306@activestate.com> Message-ID: <2mk6sz3zj1.fsf@starship.python.net> Trent Mick <trentm@ActiveState.com> writes: > Phillip J. Eby wrote: >> I've dabbled in the guts of latex2html before; it's certainly not pretty. >> IMO a better long term option might be to use the Python docutils >> and migrate to reStructuredText, since there are a bevy of backends >> available for latex, PDF, HTML, etc. They would probably need more >> work before they'll be suitable to handle the entire doc generation >> process, though. > > What do people think about docbook? Where did this conversation start again? Is it just that generating HTML currently involves "Probably the longest Perl script anyone ever attempted to maintain."? Suggestions for alternative documentation formats are a common theme in Python-land; *reasons* for changing format are much harder to find... I think it is probably possible to remove Fred from the critical path of performing a Python release, because due in large part to his work, building the Python docs just isn't that difficult (heck, my latex2html install hasn't rotted and died in *years*). I'm pretty sure it's possible on the starship, too. Of course, if Fred doesn't do the doc bits, someone else has to -- I don't know if Martin or Anthony want to take on yet another task. There might be something to be said for finding someone in a more convenient timezone. Is the process of scattering files around on creosote automated? Cheers, mwh -- <shapr> ucking keyoar -- from Twisted.Quotes From ncoghlan at iinet.net.au Sat Nov 6 16:09:47 2004 From: ncoghlan at iinet.net.au (Nick Coghlan) Date: Sat Nov 6 16:09:57 2004 Subject: [Python-Dev] Re: interlocking dependencies on the path to a release In-Reply-To: <418CA3A7.709@v.loewis.de> References: <41896727.7030206@v.loewis.de> <4188BE21.6040205@interlink.com.au> <41896727.7030206@v.loewis.de> <5.1.1.6.0.20041105101943.02b94c80@mail.telecommunity.com> <418CA3A7.709@v.loewis.de> Message-ID: <418CE93B.7010605@iinet.net.au> Martin v. L?wis wrote: > Phillip J. Eby wrote: > >> I've dabbled in the guts of latex2html before; it's certainly not pretty. > > > I wouldn't look into the source of latex2html at all. Instead, I would > rewrite it from scratch, worrying only that the output stays the same > (or sufficiently similar). Being able to easily build the current docs on a Windows system would be convenient. Even a full install of Cygwin doesn't include all the tools needed to build by the current process (latex2html is the main offender - I believe it *can* be made to work with Cygwin, but it's a separate download that requires a few other tweaks, and installation of a couple more support tools) >> IMO a better long term option might be to use the Python docutils and >> migrate to reStructuredText, since there are a bevy of backends >> available for latex, PDF, HTML, etc. They would probably need more >> work before they'll be suitable to handle the entire doc generation >> process, though. > > I'm not convinced this would be a good idea. Having TeX as the primary > source allows to produce well-formatted printed documentation. A _reST to TeX converter that added the extra typesetting info might be an interesting tool. So things with simple typesetting needs can be written in _reST, while complex typesetting is still possible with all the power of TeX. (I don't see how _reST could be given the same typesetting power without losing its elegant simplicity). Cheers, Nick. -- Nick Coghlan | Brisbane, Australia Email: ncoghlan@email.com | Mobile: +61 409 573 268 From pje at telecommunity.com Sat Nov 6 17:34:55 2004 From: pje at telecommunity.com (Phillip J. Eby) Date: Sat Nov 6 17:34:09 2004 Subject: [Python-Dev] Re: interlocking dependencies on the path to a release In-Reply-To: <418CA3A7.709@v.loewis.de> References: <5.1.1.6.0.20041105101943.02b94c80@mail.telecommunity.com> <41896727.7030206@v.loewis.de> <4188BE21.6040205@interlink.com.au> <41896727.7030206@v.loewis.de> <5.1.1.6.0.20041105101943.02b94c80@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20041106112258.0251eb70@mail.telecommunity.com> At 11:12 AM 11/6/04 +0100, Martin v. L?wis wrote: >Phillip J. Eby wrote: >>IMO a better long term option might be to use the Python docutils and >>migrate to reStructuredText, since there are a bevy of backends available >>for latex, PDF, HTML, etc. They would probably need more work before >>they'll be suitable to handle the entire doc generation process, though. > >I'm not convinced this would be a good idea. Having TeX as the primary >source allows to produce well-formatted printed documentation. If what you mean is that TeX itself does better formatting than e.g. the docutils PDF backend, I would note that there are two LaTeX backends for docutils: one a generic latex backend, and the other that's supposed to generate markup suitable for the current Python documentation toolchain. From pje at telecommunity.com Sat Nov 6 17:51:20 2004 From: pje at telecommunity.com (Phillip J. Eby) Date: Sat Nov 6 17:50:34 2004 Subject: [Python-Dev] Re: interlocking dependencies on the path to a release In-Reply-To: <418CA509.2050103@v.loewis.de> References: <5.1.1.6.0.20041105145001.02dee410@mail.telecommunity.com> <5.1.1.6.0.20041105101943.02b94c80@mail.telecommunity.com> <41896727.7030206@v.loewis.de> <4188BE21.6040205@interlink.com.au> <41896727.7030206@v.loewis.de> <5.1.1.6.0.20041105101943.02b94c80@mail.telecommunity.com> <5.1.1.6.0.20041105145001.02dee410@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20041106113503.02527e40@mail.telecommunity.com> At 11:18 AM 11/6/04 +0100, Martin v. L?wis wrote: >Phillip J. Eby wrote: >>More control of what? I thought that reST was specifically designed to >>accommodate all of the Python-specific markup we're using in the latex docs. > >How do you create a module index and a "global" index in reST? By adding directives, or using interpreted text, as long as the feature is supported by a given output writer. >How do you express document inclusion (in the spirit of \input)? There's an "include" directive. I don't know what you mean by the "spirit of \input". >>As a matter of language expressiveness, as far as I can tell, reST >>accomodates marking up both short strings and long blocks, with >>application-specific markup, so I don't really understand why there >>shouldn't be a largely 1:1 mapping between the markup systems. > >It's not about source code display. It is about the other 200 >typographic features that we use in the Python documentation. I don't get what source code display has to do with it. I'm pointing out that the languages (Latex and reST) have comparable expressiveness in their markup facilities, e.g.: Latex: \foo{bar} reST: `bar`:foo Latex: \begin{foo} blah blah \end{foo} reST: .. foo:: blah blah The difference is merely that the meaning of reST's equivalents to macros and environments (i.e. "interpreted text roles" and "directives") are defined using Python code rather than Latex. Of course, a latex writer could still be used to generate latex output, if that is the preferred format for printing. By the way, please don't confuse my answering your questions here with me advocating an actual migration or conversion at this point: from the dicussion so far and my rechecking the status of the various available docutils writers, it's clear to me that the writers aren't yet mature enough to handle generating the Python documentation. But I don't currently see any issues on the *reader* end. That is, I have not yet seen any issues raised that rule out reST syntax as a documentation source format. (I'm not saying there aren't any, either, just that I haven't seen such an issue raised yet.) From martin at v.loewis.de Sat Nov 6 18:35:03 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat Nov 6 18:34:51 2004 Subject: [Python-Dev] Re: interlocking dependencies on the path to a release In-Reply-To: <5.1.1.6.0.20041106112258.0251eb70@mail.telecommunity.com> References: <5.1.1.6.0.20041105101943.02b94c80@mail.telecommunity.com> <41896727.7030206@v.loewis.de> <4188BE21.6040205@interlink.com.au> <41896727.7030206@v.loewis.de> <5.1.1.6.0.20041105101943.02b94c80@mail.telecommunity.com> <5.1.1.6.0.20041106112258.0251eb70@mail.telecommunity.com> Message-ID: <418D0B47.3010302@v.loewis.de> Phillip J. Eby wrote: > If what you mean is that TeX itself does better formatting than e.g. the > docutils PDF backend, I would note that there are two LaTeX backends for > docutils: one a generic latex backend, and the other that's supposed to > generate markup suitable for the current Python documentation toolchain. That still might not be sufficient. reST certainly uses a specific subset of TeX's capabilities when generating LaTeX, so some features of TeX may simply be unaccessible when you have to stick to reST syntax. Regards, Martin From martin at v.loewis.de Sat Nov 6 18:42:58 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat Nov 6 18:42:47 2004 Subject: [Python-Dev] Re: interlocking dependencies on the path to a release In-Reply-To: <5.1.1.6.0.20041106113503.02527e40@mail.telecommunity.com> References: <5.1.1.6.0.20041105145001.02dee410@mail.telecommunity.com> <5.1.1.6.0.20041105101943.02b94c80@mail.telecommunity.com> <41896727.7030206@v.loewis.de> <4188BE21.6040205@interlink.com.au> <41896727.7030206@v.loewis.de> <5.1.1.6.0.20041105101943.02b94c80@mail.telecommunity.com> <5.1.1.6.0.20041105145001.02dee410@mail.telecommunity.com> <5.1.1.6.0.20041106113503.02527e40@mail.telecommunity.com> Message-ID: <418D0D22.4010405@v.loewis.de> Phillip J. Eby wrote: >> How do you create a module index and a "global" index in reST? > > > By adding directives, or using interpreted text, as long as the feature > is supported by a given output writer. Ah, so you would make use of extensions; i.e. we would essentially clone reST. >> How do you express document inclusion (in the spirit of \input)? > > > There's an "include" directive. I don't know what you mean by the > "spirit of \input". If include means literal inclusion, it is in the spirit of \input. > The difference is merely that the meaning of reST's equivalents to > macros and environments (i.e. "interpreted text roles" and "directives") > are defined using Python code rather than Latex. Of course, a latex > writer could still be used to generate latex output, if that is the > preferred format for printing. I did not understand that, in order to use reST in a complex application, you have to write your own markup extensions, and then support those in all the backends you care about. If that is typical, then yes, reST should be able to represent everything we have in the documentation right now. It just means that we cannot expect reST to work *out-of-the-box*. Instead, we first need to implement the missing formatting elements for the backends we care for. That sounds like a significant project of its own. Regards, Martin From pje at telecommunity.com Sat Nov 6 19:45:00 2004 From: pje at telecommunity.com (Phillip J. Eby) Date: Sat Nov 6 19:44:15 2004 Subject: [Python-Dev] Re: interlocking dependencies on the path to a release In-Reply-To: <418D0D22.4010405@v.loewis.de> References: <5.1.1.6.0.20041106113503.02527e40@mail.telecommunity.com> <5.1.1.6.0.20041105145001.02dee410@mail.telecommunity.com> <5.1.1.6.0.20041105101943.02b94c80@mail.telecommunity.com> <41896727.7030206@v.loewis.de> <4188BE21.6040205@interlink.com.au> <41896727.7030206@v.loewis.de> <5.1.1.6.0.20041105101943.02b94c80@mail.telecommunity.com> <5.1.1.6.0.20041105145001.02dee410@mail.telecommunity.com> <5.1.1.6.0.20041106113503.02527e40@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20041106130940.035910a0@mail.telecommunity.com> At 06:42 PM 11/6/04 +0100, Martin v. L?wis wrote: >Phillip J. Eby wrote: >>>How do you create a module index and a "global" index in reST? >> >>By adding directives, or using interpreted text, as long as the feature >>is supported by a given output writer. > >Ah, so you would make use of extensions; i.e. we would essentially >clone reST. If you mean clone the Python extensions to LaTeX, then yes. >>The difference is merely that the meaning of reST's equivalents to macros >>and environments (i.e. "interpreted text roles" and "directives") are >>defined using Python code rather than Latex. Of course, a latex writer >>could still be used to generate latex output, if that is the preferred >>format for printing. > >I did not understand that, in order to use reST in a complex >application, you have to write your own markup extensions, and >then support those in all the backends you care about. Just as was done for LaTeX, yes. (E.g. the Python latex2html postprocessing step, and the Python LaTeX macros.) >If that is typical, then yes, reST should be able to represent >everything we have in the documentation right now. It just means >that we cannot expect reST to work *out-of-the-box*. Instead, we >first need to implement the missing formatting elements for the >backends we care for. That sounds like a significant project of >its own. Potentially so. However, there are some significant shortcuts possible for many situations. reST includes an unparameterized macro facility, a 'raw' directive that passes markup through to the backend, and a 'role' directive to create markup roles for use with interpreted text. These roles apply "class" attributes to the DOM, that then end up as HTML or XML "class" attributes, or potentially as LaTeX macros or environments. For example, one might use: .. role:: var(emphasis) to define a text role that could be used to replace \var{foo} with :var:`foo`. So, I think there are some regularities that can be exploited, such that it's not an N*M project, for N features and M backends. Those features that can be implemented in terms of existing reST concepts (other than the 'raw' passthru) would only need implementation once. Indeed, if reST simply had a suitable directive for defining directives, I think that all of the merely typographical aspects of Python documentation markup could be defined in reST, as part of the documentation source. The main main areas of "missing" features in docutils backends that I'm aware of would fall under the headings of index generation (all backends), and multi-page HTML output with navigation links (HTML backend). From pje at telecommunity.com Sat Nov 6 19:48:44 2004 From: pje at telecommunity.com (Phillip J. Eby) Date: Sat Nov 6 19:47:58 2004 Subject: [Python-Dev] Re: interlocking dependencies on the path to a release In-Reply-To: <418D0B47.3010302@v.loewis.de> References: <5.1.1.6.0.20041106112258.0251eb70@mail.telecommunity.com> <5.1.1.6.0.20041105101943.02b94c80@mail.telecommunity.com> <41896727.7030206@v.loewis.de> <4188BE21.6040205@interlink.com.au> <41896727.7030206@v.loewis.de> <5.1.1.6.0.20041105101943.02b94c80@mail.telecommunity.com> <5.1.1.6.0.20041106112258.0251eb70@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20041106134543.020f4c30@mail.telecommunity.com> At 06:35 PM 11/6/04 +0100, Martin v. L?wis wrote: >Phillip J. Eby wrote: >>If what you mean is that TeX itself does better formatting than e.g. the >>docutils PDF backend, I would note that there are two LaTeX backends for >>docutils: one a generic latex backend, and the other that's supposed to >>generate markup suitable for the current Python documentation toolchain. > >That still might not be sufficient. reST certainly uses a specific subset >of TeX's capabilities when generating LaTeX, so some features >of TeX may simply be unaccessible when you have to stick to reST >syntax. There's a 'raw' directive that can be used for this: .. raw:: latex \setlength{\parindent}{0pt} The data will be ignored by non-latex backends. Similarly, you may use raw HTML with 'raw:: html'. See http://docutils.sourceforge.net/docs/ref/rst/directives.html#raw-data-pass-through From jcarlson at uci.edu Sun Nov 7 00:05:44 2004 From: jcarlson at uci.edu (Josiah Carlson) Date: Sun Nov 7 00:14:59 2004 Subject: [Python-Dev] Synchronous and Asynchronous servers in the standard library Message-ID: <20041106102107.F853.JCARLSON@uci.edu> A recent patch to offer an SMTP server (SocketServer derivative) sparked the below... Question: Does we (and by 'we' I mean those in charge of developing Python) want to offer both asynchronous (deriving from asyncore, asynchat, etc.) and synchronous versions of server software in the standard library? Observations: There is currently 1 async server, and 5 sync servers in the standard library. If we want to offer both sync and async versions, it makes sense to reduce programmer effort and share as much code as possible. Idea: Set up a framework that allows Python programmers to produce async and sync variants of their servers easily. I have a partial example 'implementation' below. It seems to make sense (if such a framework is desired) to place all such modules in a 'Server' package, and to describe precisely what is necessary to make such a package usable, used, and added to in a PEP called something like "A flexible server framework". Future Python: This would obviously be a 2.5 feature/module, and I would be willing to write both the PEP (perhaps after the 1st of January, 2005 as I have a journal paper write before then) (and afterwards) translate the current standard library servers to the framework. Depending on how popular the framework is through the lifetime of Python 2.5 (if it makes it into 2.5), it may make sense to deprecate the original servers in 2.6 or 2.7 (if ever). I would be willing to volunteer to maintain the server package, or at least as many of the protocols as I understand, for 2.5 and beyond (I don't know much about XMLRPC, so translation is about as much as I could probably do). - Josiah In the case of 'chatty' protocols like SMTP, POP, etc., where the following complete, merely a prototype... class SyncServerBase: def send(self, data): if data is None: self.sock.close() else: self.sock.write(data) def close_when_done(self): self.send(None) ... class AsyncServerBase(asyncore.dispatcher): def send(self, data): #handle buffering with deque, etc. def close_when_done(self): #toss a None into the queue ... class AsynchatMixin(asynchat.async_chat): #perhaps write a better buffered reader and writer... limit = None def readable(self): #handle optional request size limit return limit is None or len(self.ac_in_buffer) < self.limit def found_terminator(self): self.request = self.ac_in_buffer self.ac_in_buffer = '' self.handle_request() ... class SMTP_Handler: def SMTP_HELO(self, cmd): .... def SMTP_MAIL(self, cmd): ... self.send(response) ... ... class SMTP_Sync(SyncServerBase, SMTP_Handler): def handle_request(self): #does all of the reading necessary to read from a socket and #handle dispatch to SMTP_* class SMTP_Async(AsyncServerBase, AsynchatMixin, SMTP_Handler): terminator = '\r\n' def handle_request(self): #command line (or email content) is provided in self.request #handles dispatch to SMTP_*, reading a line/request is provided #in this case by the AsynchatMixin Etcetera. From tim.peters at gmail.com Sun Nov 7 04:31:33 2004 From: tim.peters at gmail.com (Tim Peters) Date: Sun Nov 7 04:31:41 2004 Subject: [Python-Dev] patch-finalizer vs Zope3 In-Reply-To: <1f7befae041105120727ae399e@mail.gmail.com> References: <20041104171957.A0C633B8038@smtp.zope.com> <1f7befae041105120727ae399e@mail.gmail.com> Message-ID: <1f7befae04110619317dfe0f4d@mail.gmail.com> [Tim, on a patch-finalizer consequence] > OK, that actually wasn't the first test to produce garbage, it was > just the first time test.py *noticed* gc.garbage wasn't empty. The > first test to produce garbage was BuddyCityState, and it produces > garbage in isolation: > ... Man, this sure brings back miserable memories of how hard it could be to debug leaks due to cycles before Neil added gcmodule.c! The best tool we had then was my Cyclops.py, but I stopped keeping that up to date because there was obviously no more need for it <wink>. Desktop boxes have gotten a lot more capable since then, and apps have grown in ambition to match. Here we ran one measly test from the huge Zope3 test suite; it ran in an eye blink (maybe two). From the first weakref in gc.garbage alone, it turned out that more than 42,000 distinct cycles were reachable, breaking into a bit fewer than 4,000 strongly connected components, the latter ranging in size from 2 objects (almost all a new-style class and its mro) to about 40,000. A text file account of these SCCs consumed about 20MB. So what you do? I don't know. These are always solvable, but they're rarely easy. I always end up writing a small pile of new graph analysis code specifically exploiting peculiarities of the case at hand. In this case I also changed gcmodule.c a little, to put trash reachable from resurrected trash weakrefs into gc.garbage too, but unlike gc.DEBUG_SAVEALL not to also put *all* trash in gc.garbage. That was key to figuring out what the original trash cycles were. "The answer" in this case appears to be in Zope3's zope/interface/adapter.py's AdapterRegistry.__init__: class AdapterRegistry(object): """Adapter registry """ # Implementation note: # We are like a weakref dict ourselves. We can't use a weakref # dict because we have to use spec.weakref() rather than # weakref.ref(spec) to get weak refs to specs. _surrogateClass = Surrogate def __init__(self): default = self._surrogateClass(Default, self) self._default = default null = self._surrogateClass(Null, self) self._null = null # Create separate lookup object and copy it's methods surrogates = {Default.weakref(): default, Null.weakref(): null} def _remove(k): try: del surrogates[k] except KeyError: pass lookup = AdapterLookup(self, surrogates, _remove) The rest doesn't appear to matter. As the attached smoking gun shows, a weakref W has an instance of the nested _remove function as its callback(*). The _remove function has a cell object, pointing to the lexically enclosing `surrogates` dict. That dict in turn has W as a key. So there's a cycle, and refcounting alone can't clean it up. All of this stuff *was* trash, but because patch-finalizer decides to call W reachable, the cell and the `surrogates` dict became reachable too, and none of it got cleaned up (nor, of course, did anything else reachable from this cycle). When 2.4b2 looks at this case, it clears W because its referent is trash, and ignore's W's callback because W is also trash. Since, had it been called, the only thing the callback would have done is mutate other trash, it didn't hurt not to call it. But it remains a stretch to believe that "ignoring callbacks on trash weakrefs is harmless" is a necessary outcome. Still, whatever else this may or may not imply, it convinces me we can't adopt a patch-finalizer-ish approach without serious effort at making leaks "due to it" more easily analyzable. (*) A weakref W's callback is discoverable after all in Python, just by calling gc.get_referents(W). This is something I've rediscovered about 6 times by now, and it always catches me by surprise <wink>. -------------- next part -------------- cycle obj 0: cycle starts here 0x3c84f20 <type 'weakref'> <weakref at 03C84F20; to 'InterfaceClass' at 012B8118 (IWriteContainer)> weakref to <InterfaceClass zope.app.container.interfaces.IWriteContainer> obj 1: 0x3c89a30 <type 'function'> <function _remove at 0x03C89A30> obj 2: 0x3c8b658 <type 'tuple'> (<cell at 0x03C8B968: dict object at 0x03C6A620>,) obj 3: 0x3c8b968 <type 'cell'> <cell at 0x03C8B968: dict object at 0x03C6A620> obj 4: 0x3c6a620 <type 'dict'> {<weakref at 020628F0; to 'InterfaceClass' at 015748F8 (Default)>: <Surrogate(<InterfaceClass zope.interface.adapter.Default>)>, <weakref at 020629C8; to 'InterfaceClass' at 01574A10 (Null)>: <Surrogate(<InterfaceClass zope.interface.adapter.Null>)>, <weakref at 03C84F20; to 'InterfaceClass' at 012B8118 (IWriteContainer)>: <Surrogate(<InterfaceClass zope.app.container.interfaces.IWriteContainer>)>, <weakref at 03C8C080; to 'InterfaceClass' at 00BD0188 (Interface)>: <Surrogate(<InterfaceClass zope.interface.Interface>)>, <weakref at 03C8C278; to 'InterfaceClass' at 012B3D20 (IObjectEvent)>: <Surrogate(<InterfaceClass zope.app.event.interfaces.IObjectEvent>)>, <weakref at 03C8C3E0; to 'InterfaceClass' at 011909D8 (IHTTPRequest)>: <Surrogate(<InterfaceClass zope.publisher.interfaces.http.IHTTPRequest>)>, <weakref at 03C8C500; to 'InterfaceClass' at 011904D0 (IRequest)>: <Surrogate(<InterfaceClass zope.publisher.interfaces.IRequest>)>, <weakref at 03C8C620; to 'InterfaceClass' at 01185FC0 (IPublisherRequest)>: <Surrogate(<InterfaceClass zope.publisher.interfaces.IPublisherRequest>)>, <weakref at 03C8C740; to 'InterfaceClass' at 01185D90 (IPublicationRequest)>: <Surrogate(<InterfaceClass zope.publisher.interfaces.IPublicationRequest>)>, <weakref at 03C8C860; to 'InterfaceClass' at 00DB5D20 (IPresentationRequest)>: <Surrogate(<InterfaceClass zope.component.interfaces.IPresentationRequest>)>, <weakref at 03C8C9C8; to 'InterfaceClass' at 00D99818 (IParticipation)>: <Surrogate(<InterfaceClass zope.security.interfaces.IParticipation>)>, <weakref at 03C8CC08; to 'InterfaceClass' at 011903F0 (IApplicationRequest)>: <Surrogate(<InterfaceClass zope.publisher.interfaces.IApplicationRequest>)>, <weakref at 03C8CD28; to 'InterfaceClass' at 0117A700 (IEnumerableMapping)>: <Surrogate(<InterfaceClass zope.interface.common.mapping.IEnumerableMapping>)>, <weakref at 03C8CE48; to 'InterfaceClass' at 0117A540 (IReadMapping)>: <Surrogate(<InterfaceClass zope.interface.common.mapping.IReadMapping>)>, <weakref at 03C8CF68; to 'InterfaceClass' at 0117A428 (IItemMapping)>: <Surrogate(<InterfaceClass zope.interface.common.mapping.IItemMapping>)>} cycle ends here 0x3c84f20 <type 'weakref'> <weakref at 03C84F20; to 'InterfaceClass' at 012B8118 (IWriteContainer)> weakref to <InterfaceClass zope.app.container.interfaces.IWriteContainer> From tim.peters at gmail.com Sun Nov 7 06:07:37 2004 From: tim.peters at gmail.com (Tim Peters) Date: Sun Nov 7 06:07:39 2004 Subject: [Python-Dev] patch-finalizer vs Zope3 In-Reply-To: <1f7befae04110619317dfe0f4d@mail.gmail.com> References: <20041104171957.A0C633B8038@smtp.zope.com> <1f7befae041105120727ae399e@mail.gmail.com> <1f7befae04110619317dfe0f4d@mail.gmail.com> Message-ID: <1f7befae04110621071e42e173@mail.gmail.com> [Tim] ... > surrogates = {Default.weakref(): default, Null.weakref(): null} > def _remove(k): > try: > del surrogates[k] > except KeyError: > pass FYI, if I fiddle that part to use the same kind of odd spelling Python's weak dicts stumbled into for other reasons: surrogates = {Default.weakref(): default, Null.weakref(): null} self._surrogates = surrogates # new def _remove(k, selfref=weakref.ref(self)): # changed self = selfref() # new if self is not None: # new try: del self._surrogates[k] # changed except KeyError: pass then the Zope3 unit tests don't leave anything in gc.garbage under 2.4b2 + patch-finalizer. A spattering of tests fail (2 failures + 3 errors), though, due to new TypeError: can't pickle weakref objects exceptions. Note that in this case the callbacks get invoked but are impotent: self isn't strongly reachable from the callbacks, so self is left in the unreachable set, and so the `selfref` weakref (which does not have a callback) gets cleared before gc starts breaking cycles: all the callbacks see selfref() return None. From astrand at lysator.liu.se Sun Nov 7 11:20:47 2004 From: astrand at lysator.liu.se (Peter Astrand) Date: Sun Nov 7 11:20:54 2004 Subject: [Python-Dev] Re: Bug [ 959379 ] Implicit close() should check for errors In-Reply-To: <417EAD88.7080105@v.loewis.de> References: <Pine.GSO.4.51L2.0410242311270.5858@koeberg.lysator.liu.se> <ulldubi9v.fsf@fitlinxx.com> <Pine.GSO.4.51L2.0410260750530.29046@koeberg.lysator.liu.se> <417E81B1.8080601@v.loewis.de> <Pine.GSO.4.51L2.0410262044200.14637@koeberg.lysator.liu.se> <417EAD88.7080105@v.loewis.de> Message-ID: <Pine.GSO.4.51L2.0411071118080.14526@koeberg.lysator.liu.se> On Tue, 26 Oct 2004, "Martin v. L?wis" wrote: > Peter Astrand wrote: > > + PySys_WriteStderr("close failed: %s\n", strerror(errno)); > > This should use HAVE_STRERROR. Like this?: diff -u -r2.192 fileobject.c --- dist/src/Objects/fileobject.c 11 Jun 2004 04:49:03 -0000 2.192 +++ dist/src/Objects/fileobject.c 7 Nov 2004 10:13:04 -0000 @@ -300,12 +300,19 @@ static void file_dealloc(PyFileObject *f) { + int sts = 0; if (f->weakreflist != NULL) PyObject_ClearWeakRefs((PyObject *) f); if (f->f_fp != NULL && f->f_close != NULL) { Py_BEGIN_ALLOW_THREADS - (*f->f_close)(f->f_fp); + sts = (*f->f_close)(f->f_fp); Py_END_ALLOW_THREADS + if (sts == EOF) +#ifdef HAVE_STRERROR + PySys_WriteStderr("close failed: [Errno %d] %s\n", errno, strerror(errno)); +#else + PySys_WriteStderr("close failed: [Errno %d]\n", errno); +#endif } PyMem_Free(f->f_setbuf); Py_XDECREF(f->f_name); /Peter ?strand <astrand@lysator.liu.se> From anthony at interlink.com.au Sun Nov 7 15:32:35 2004 From: anthony at interlink.com.au (Anthony Baxter) Date: Sun Nov 7 15:32:53 2004 Subject: [Python-Dev] Synchronous and Asynchronous servers in the standard library In-Reply-To: <20041106102107.F853.JCARLSON@uci.edu> References: <20041106102107.F853.JCARLSON@uci.edu> Message-ID: <418E3203.9060002@interlink.com.au> Josiah Carlson wrote: > A recent patch to offer an SMTP server (SocketServer derivative) sparked > the below... > > Question: > Does we (and by 'we' I mean those in charge of developing Python) want > to offer both asynchronous (deriving from asyncore, asynchat, etc.) and > synchronous versions of server software in the standard library? Personally, I think that encouraging anyone to develop new software on top of asyncore/asynchat is a _terrible_ idea. Anthony From jhylton at gmail.com Sun Nov 7 16:09:34 2004 From: jhylton at gmail.com (Jeremy Hylton) Date: Sun Nov 7 16:09:40 2004 Subject: [Python-Dev] Synchronous and Asynchronous servers in the standard library In-Reply-To: <418E3203.9060002@interlink.com.au> References: <20041106102107.F853.JCARLSON@uci.edu> <418E3203.9060002@interlink.com.au> Message-ID: <e8bf7a5304110707093414cfa3@mail.gmail.com> On Mon, 08 Nov 2004 01:32:35 +1100, Anthony Baxter <anthony@interlink.com.au> wrote: > Josiah Carlson wrote: > > A recent patch to offer an SMTP server (SocketServer derivative) sparked > > the below... > > > > Question: > > Does we (and by 'we' I mean those in charge of developing Python) want > > to offer both asynchronous (deriving from asyncore, asynchat, etc.) and > > synchronous versions of server software in the standard library? > > Personally, I think that encouraging anyone to develop new software on > top of asyncore/asynchat is a _terrible_ idea. I agree. Jeremy From allison at sumeru.stanford.EDU Sun Nov 7 17:11:20 2004 From: allison at sumeru.stanford.EDU (Dennis Allison) Date: Sun Nov 7 17:11:38 2004 Subject: [Python-Dev] Synchronous and Asynchronous servers in the standard library In-Reply-To: <e8bf7a5304110707093414cfa3@mail.gmail.com> Message-ID: <Pine.LNX.4.10.10411070805390.25017-100000@sumeru.stanford.EDU> The question, then, is what base should be used for developing asynchonous servers? On Sun, 7 Nov 2004, Jeremy Hylton wrote: > On Mon, 08 Nov 2004 01:32:35 +1100, Anthony Baxter > <anthony@interlink.com.au> wrote: > > Josiah Carlson wrote: > > > A recent patch to offer an SMTP server (SocketServer derivative) sparked > > > the below... > > > > > > Question: > > > Does we (and by 'we' I mean those in charge of developing Python) want > > > to offer both asynchronous (deriving from asyncore, asynchat, etc.) and > > > synchronous versions of server software in the standard library? > > > > Personally, I think that encouraging anyone to develop new software on > > top of asyncore/asynchat is a _terrible_ idea. > > I agree. > > Jeremy From foom at fuhm.net Sun Nov 7 17:13:59 2004 From: foom at fuhm.net (James Y Knight) Date: Sun Nov 7 17:14:05 2004 Subject: [Python-Dev] Synchronous and Asynchronous servers in the standard library In-Reply-To: <20041106102107.F853.JCARLSON@uci.edu> References: <20041106102107.F853.JCARLSON@uci.edu> Message-ID: <087A5D6A-30D8-11D9-9BCD-000A95A50FB2@fuhm.net> On Nov 6, 2004, at 6:05 PM, Josiah Carlson wrote: > A recent patch to offer an SMTP server (SocketServer derivative) > sparked > the below... > > Question: > Does we (and by 'we' I mean those in charge of developing Python) want > to offer both asynchronous (deriving from asyncore, asynchat, etc.) and > synchronous versions of server software in the standard library? I'd like to introduce you to Twisted (http://www.twistedmatrix.com). More cooperation and code-sharing between Twisted and Python core developers would certainly be nice. But I don't think the right direction to go for that is writing a bunch of half-assed async servers in Python core. I guarantee you it isn't going to be trivial to do nicely. There has been a small amount of discussion about contributing parts of the Twisted core into Python, but so far no one has volunteered to head that project. So, if you're interested, perhaps you'd like to take it on. I suspect even that will be a lot of work -- you'd have to get consensus on which parts, if any, would be included. Figure out acceptable solutions to versioning skew between twisted releases and python's included copy. Etc. James From arigo at tunes.org Sun Nov 7 17:27:10 2004 From: arigo at tunes.org (Armin Rigo) Date: Sun Nov 7 17:51:29 2004 Subject: [Python-Dev] segfault/Py_FatalError at exit with threads Message-ID: <20041107162710.GA26684@vicky.ecs.soton.ac.uk> Hi, Someone familiar with pystate.c and PyGILState_*() should assign himself to bug http://www.python.org/sf/1061968 -- I don't know all the intricaties of the locking mecanisms to fix it myself... Armin From carribeiro at gmail.com Sun Nov 7 18:00:26 2004 From: carribeiro at gmail.com (Carlos Ribeiro) Date: Sun Nov 7 18:00:44 2004 Subject: [Python-Dev] Re: PEP 336, "Make None Callable", by Andrew McClelland In-Reply-To: <41899E5D.2080303@interlink.com.au> References: <E1CPOSu-0005MB-RF@sc8-pr-cvs1.sourceforge.net> <41899E5D.2080303@interlink.com.au> Message-ID: <864d3709041107090070849085@mail.gmail.com> On Thu, 04 Nov 2004 14:13:33 +1100, Anthony Baxter <anthony@interlink.com.au> wrote: > > > Abstract > > > > None should be a callable object that when called with any > > arguments has no side effect and returns None. > > My response to this is simply "yuck". This is a hack, abd I > don't think it's worthwhile. If you want to test for None, test > for None. Similarly, if you want a dictionary lookup to default > to a no-op method, there's a perfectly simple way to spell it now: > > def defaultNoopMethod(*args, **kwargs): pass > somedict.get(key, defaultNoopMethod) > > You can even do this as a one-liner using a lambda if you want. Just a question, and I sincerely hope it's not a dumb one. Python already have a noop statement ("pass"). Now, what is being requested is a standard noop callable. Does it make sense -- both in theorethical and practical terms -- to allow a statement such as "pass" to be used in both situations? In other words, does it make sense to have a "unification" of sorts? Thanks for any pointers, -- Carlos Ribeiro Consultoria em Projetos blog: http://rascunhosrotos.blogspot.com blog: http://pythonnotes.blogspot.com mail: carribeiro@gmail.com mail: carribeiro@yahoo.com From martin at v.loewis.de Sun Nov 7 19:08:08 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun Nov 7 19:08:11 2004 Subject: [Python-Dev] Re: Bug [ 959379 ] Implicit close() should check for errors In-Reply-To: <Pine.GSO.4.51L2.0411071118080.14526@koeberg.lysator.liu.se> References: <Pine.GSO.4.51L2.0410242311270.5858@koeberg.lysator.liu.se> <ulldubi9v.fsf@fitlinxx.com> <Pine.GSO.4.51L2.0410260750530.29046@koeberg.lysator.liu.se> <417E81B1.8080601@v.loewis.de> <Pine.GSO.4.51L2.0410262044200.14637@koeberg.lysator.liu.se> <417EAD88.7080105@v.loewis.de> <Pine.GSO.4.51L2.0411071118080.14526@koeberg.lysator.liu.se> Message-ID: <418E6488.6060804@v.loewis.de> Peter Astrand wrote: > Like this?: Yes, look good - please check it in. Martin From martin at v.loewis.de Sun Nov 7 19:20:50 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun Nov 7 19:20:53 2004 Subject: [Python-Dev] Re: PEP 336, "Make None Callable", by Andrew McClelland In-Reply-To: <864d3709041107090070849085@mail.gmail.com> References: <E1CPOSu-0005MB-RF@sc8-pr-cvs1.sourceforge.net> <41899E5D.2080303@interlink.com.au> <864d3709041107090070849085@mail.gmail.com> Message-ID: <418E6782.6060004@v.loewis.de> Carlos Ribeiro wrote: > Just a question, and I sincerely hope it's not a dumb one. Python > already have a noop statement ("pass"). Now, what is being requested > is a standard noop callable. Does it make sense -- both in > theorethical and practical terms -- to allow a statement such as > "pass" to be used in both situations? In other words, does it make > sense to have a "unification" of sorts? No. Theoretically, the "pass" statement exists for purely syntactical purposes: namely, to put a statement where a statement is required, but no meaningful statement can be found. As a result, "pass" has no run-time semantics: it is not the case that it is executed and does nothing, instead, it is simply not executed. The proposed "function with no effect" would be quite different: it would have a run-time semantics, which would be to ignore all arguments, and return None. Practically, it would be very confusing if the keyword "pass" suddenly started to denote a function. Traditionally, keywords have never denoted expressions. It might be that True, False, and None become keywords some day, but these would *only* be use as expressions. It is unpythonic to give an expression meaning to a statement (just as assignment is not an expression, either). The simplest solution to achieve the rationale of the PEP would be to add a function operator.nop. Of course, writing def nop(*args):pass isn't that more difficult than writing from operator import nop Regards, Martin From jcarlson at uci.edu Sun Nov 7 19:39:43 2004 From: jcarlson at uci.edu (Josiah Carlson) Date: Sun Nov 7 19:47:33 2004 Subject: [Python-Dev] Synchronous and Asynchronous servers in the standard library In-Reply-To: <087A5D6A-30D8-11D9-9BCD-000A95A50FB2@fuhm.net> References: <20041106102107.F853.JCARLSON@uci.edu> <087A5D6A-30D8-11D9-9BCD-000A95A50FB2@fuhm.net> Message-ID: <20041107093047.F85C.JCARLSON@uci.edu> James Y Knight <foom@fuhm.net> wrote: > > On Nov 6, 2004, at 6:05 PM, Josiah Carlson wrote: > > > A recent patch to offer an SMTP server (SocketServer derivative) > > sparked > > the below... > > > > Question: > > Does we (and by 'we' I mean those in charge of developing Python) want > > to offer both asynchronous (deriving from asyncore, asynchat, etc.) and > > synchronous versions of server software in the standard library? > > I'd like to introduce you to Twisted (http://www.twistedmatrix.com). > > More cooperation and code-sharing between Twisted and Python core > developers would certainly be nice. But I don't think the right > direction to go for that is writing a bunch of half-assed async servers > in Python core. I guarantee you it isn't going to be trivial to do > nicely. There has been a small amount of discussion about contributing > parts of the Twisted core into Python, but so far no one has > volunteered to head that project. So, if you're interested, perhaps > you'd like to take it on. I suspect even that will be a lot of work -- > you'd have to get consensus on which parts, if any, would be included. > Figure out acceptable solutions to versioning skew between twisted > releases and python's included copy. Etc. It seems as though there has already been discussion of getting portions of Twisted into the Python standard library. Being as I have not developed with Twisted, nor am I really a part of the Twisted community, I'm not sure that I am really the right person to try to figure out what parts of Twisted should or should not be included with Python 2.5. Regardless of the asyncore vs Twisted issue, a proper (meta) framework should separate out the calls for "what happens to the internal state when the server receives a reqest like 'foo'", and how the data actually gets to those calls. If done correctly, if/when portions of Twisted make it into the standard library, it shouldn't be terribly difficult to offer a mechanism to use Twisted as a base (for someone with Twisted and stdlib Twisted experience), if such a thing is desired. I suppose the question comes down to; should there be a meta framework for developing servers in Python that specifies a separation between the protocol logic and how data gets to that protocol logic? The desire is for a mechanism to allow people to create a single version of a server in their favorite framework (SocketServer, asyncore, Twisted, or other), and for others to be able to use them in their favorite framework with little difficulty. - Josiah From pje at telecommunity.com Sun Nov 7 20:15:57 2004 From: pje at telecommunity.com (Phillip J. Eby) Date: Sun Nov 7 20:15:21 2004 Subject: [Python-Dev] Synchronous and Asynchronous servers in the standard library In-Reply-To: <20041107093047.F85C.JCARLSON@uci.edu> References: <087A5D6A-30D8-11D9-9BCD-000A95A50FB2@fuhm.net> <20041106102107.F853.JCARLSON@uci.edu> <087A5D6A-30D8-11D9-9BCD-000A95A50FB2@fuhm.net> Message-ID: <5.1.1.6.0.20041107135655.020fbd00@mail.telecommunity.com> At 10:39 AM 11/7/04 -0800, Josiah Carlson wrote: >It seems as though there has already been discussion of getting portions >of Twisted into the Python standard library. Being as I have not >developed with Twisted, nor am I really a part of the Twisted community, >I'm not sure that I am really the right person to try to figure out what >parts of Twisted should or should not be included with Python 2.5. > >Regardless of the asyncore vs Twisted issue, a proper (meta) framework >should separate out the calls for "what happens to the internal state >when the server receives a reqest like 'foo'", and how the data actually >gets to those calls. If done correctly, if/when portions of Twisted >make it into the standard library, it shouldn't be terribly difficult to >offer a mechanism to use Twisted as a base (for someone with Twisted and >stdlib Twisted experience), if such a thing is desired. > >I suppose the question comes down to; should there be a meta framework >for developing servers in Python that specifies a separation between the >protocol logic and how data gets to that protocol logic? The desire is >for a mechanism to allow people to create a single version of a server >in their favorite framework (SocketServer, asyncore, Twisted, or other), >and for others to be able to use them in their favorite framework with >little difficulty. I believe that this may be an unrealistic goal, even though I have done something rather similar myself. There is a myriad assortment of policy issues you'll have to swim upstream past, just to get to the mechanism issues. For example, Twisted assumes its reactor is an interpreter-global, maybe even process-global event loop, whereas asyncore and peak.events allow multiple event loops. I've dabbled in integrating Twisted and peak.events, such that code written for peak.events can run with or without Twisted, and such that an asynchronous server can be run synchronously. So, it's certainly possible to do something like what you're describing, but I'm not sure it's worth the effort versus someone simply installing PEAK and/or Twisted. Now, if the Python standard library included a lightweight task switching facility akin to Armin's greenlets, it might be more practical to include asynchronous communications facilities in the stdlib. Essentially, one could implement a set of communication primitives that task-switched to a global scheduler, or one could supply the scheduler as part of the calls, or provide the primitives as methods of a scheduler, etc. This would bypass most of the architectural policy issues. However, without something like this, you are basically just going to be re-inventing either Twisted or peak.events, depending on whether you prefer a continuation-passing or generator-pseudothread architecture. From bob at redivi.com Sun Nov 7 20:19:09 2004 From: bob at redivi.com (Bob Ippolito) Date: Sun Nov 7 20:19:17 2004 Subject: [Python-Dev] Synchronous and Asynchronous servers in the standard library In-Reply-To: <20041107093047.F85C.JCARLSON@uci.edu> References: <20041106102107.F853.JCARLSON@uci.edu> <087A5D6A-30D8-11D9-9BCD-000A95A50FB2@fuhm.net> <20041107093047.F85C.JCARLSON@uci.edu> Message-ID: <E6A43674-30F1-11D9-9D1F-000A95BA5446@redivi.com> On Nov 7, 2004, at 13:39, Josiah Carlson wrote: > I suppose the question comes down to; should there be a meta framework > for developing servers in Python that specifies a separation between > the > protocol logic and how data gets to that protocol logic? The desire is > for a mechanism to allow people to create a single version of a server > in their favorite framework (SocketServer, asyncore, Twisted, or > other), > and for others to be able to use them in their favorite framework with > little difficulty. That's not really practical. You can't use synchronous protocol code in an asynchronous server without threads or Stackless... so what's the point? As for asynchronous frameworks that are compatible with the Python license, I'm only aware of asyncore (which people generally feel should be deprecated) and Twisted. I think the best course of action would be to create some stripped down derivative of Twisted. A couple changes off the top of my head would make it more suitable for the standard library: - Allow generators to be easily used for asynchrony (much easier to write synchronous-like code in that style, this has been demonstrated by several people) - Throw a blocking API on top because some people will want to use it in a blocking manner (Twisted's testing framework already has this functionality) - Allow more than one reactor per process (necessary to support the previous, because invariably people will use the blocking API from several threads) - Bring in some of the standard protocols (SMTP, HTTP, ...) but not the higher-level versions when available (twisted.web) because those are so much more volatile. -bob From jcarlson at uci.edu Sun Nov 7 22:37:10 2004 From: jcarlson at uci.edu (Josiah Carlson) Date: Sun Nov 7 22:44:36 2004 Subject: [Python-Dev] Synchronous and Asynchronous servers in the standard library In-Reply-To: <E6A43674-30F1-11D9-9D1F-000A95BA5446@redivi.com> References: <20041107093047.F85C.JCARLSON@uci.edu> <E6A43674-30F1-11D9-9D1F-000A95BA5446@redivi.com> Message-ID: <20041107132901.F868.JCARLSON@uci.edu> Bob Ippolito <bob@redivi.com> wrote: > That's not really practical. You can't use synchronous protocol code > in an asynchronous server without threads or Stackless... so what's the > point? As for asynchronous frameworks that are compatible with the > Python license, I'm only aware of asyncore (which people generally feel > should be deprecated) and Twisted. At least in the standard library right now, there is really one major difference between sync and async servers; how data gets to and from functions that implement the logic. > I think the best course of action would be to create some stripped down > derivative of Twisted. A couple changes off the top of my head would > make it more suitable for the standard library: > > - Allow generators to be easily used for asynchrony (much easier to > write synchronous-like code in that style, this has been demonstrated > by several people) > - Throw a blocking API on top because some people will want to use it > in a blocking manner (Twisted's testing framework already has this > functionality) > - Allow more than one reactor per process (necessary to support the > previous, because invariably people will use the blocking API from > several threads) > - Bring in some of the standard protocols (SMTP, HTTP, ...) but not > the higher-level versions when available (twisted.web) because those > are so much more volatile. This all sounds reasonable to me, though I am certain that I am not the right man for doing such a conversion. - Josiah From jcarlson at uci.edu Sun Nov 7 22:53:56 2004 From: jcarlson at uci.edu (Josiah Carlson) Date: Sun Nov 7 23:01:36 2004 Subject: [Python-Dev] Synchronous and Asynchronous servers in the standard library In-Reply-To: <5.1.1.6.0.20041107135655.020fbd00@mail.telecommunity.com> References: <20041107093047.F85C.JCARLSON@uci.edu> <5.1.1.6.0.20041107135655.020fbd00@mail.telecommunity.com> Message-ID: <20041107133719.F86E.JCARLSON@uci.edu> "Phillip J. Eby" <pje@telecommunity.com> wrote: > > I believe that this may be an unrealistic goal, even though I have done > something rather similar myself. There is a myriad assortment of policy > issues you'll have to swim upstream past, just to get to the mechanism > issues. For example, Twisted assumes its reactor is an interpreter-global, > maybe even process-global event loop, whereas asyncore and peak.events > allow multiple event loops. > > I've dabbled in integrating Twisted and peak.events, such that code written > for peak.events can run with or without Twisted, and such that an > asynchronous server can be run synchronously. So, it's certainly possible > to do something like what you're describing, but I'm not sure it's worth > the effort versus someone simply installing PEAK and/or Twisted. Since most people are saying that this would be ugly (at least in the realm of Twisted), I'll take back the offer. > Now, if the Python standard library included a lightweight task switching > facility akin to Armin's greenlets, it might be more practical to include > asynchronous communications facilities in the stdlib. Essentially, one > could implement a set of communication primitives that task-switched to a > global scheduler, or one could supply the scheduler as part of the calls, > or provide the primitives as methods of a scheduler, etc. This would > bypass most of the architectural policy issues. Re: greenlets If someone were ambitious, one could write an implementation of greenlets using standard threads for platforms without an optimized greenlets implementation. - Josiah From allison at sumeru.stanford.EDU Mon Nov 8 00:33:35 2004 From: allison at sumeru.stanford.EDU (Dennis Allison) Date: Mon Nov 8 00:33:45 2004 Subject: [Python-Dev] Synchronous and Asynchronous servers in the standard library In-Reply-To: <20041107133719.F86E.JCARLSON@uci.edu> Message-ID: <Pine.LNX.4.10.10411071531210.25017-100000@sumeru.stanford.EDU> It might be worthwhile to list what the existing problems are with the asyncore and asynchat libraries. There is also the problem that many applications, Zope among them, depend upon them. From andrew-pythondev at puzzling.org Mon Nov 8 01:19:01 2004 From: andrew-pythondev at puzzling.org (Andrew Bennetts) Date: Mon Nov 8 01:19:27 2004 Subject: [Python-Dev] Synchronous and Asynchronous servers in the standard library In-Reply-To: <Pine.LNX.4.10.10411071531210.25017-100000@sumeru.stanford.EDU> References: <20041107133719.F86E.JCARLSON@uci.edu> <Pine.LNX.4.10.10411071531210.25017-100000@sumeru.stanford.EDU> Message-ID: <20041108001901.GB477@frobozz> On Sun, Nov 07, 2004 at 03:33:35PM -0800, Dennis Allison wrote: > > It might be worthwhile to list what the existing problems are with the > asyncore and asynchat libraries. There is also the problem that many > applications, Zope among them, depend upon them. One such list is here: http://mail.zope.org/pipermail/zope3-dev/2002-October/003235.html -Andrew. From martin at v.loewis.de Mon Nov 8 06:50:25 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon Nov 8 06:50:27 2004 Subject: [Python-Dev] Synchronous and Asynchronous servers in the standard library In-Reply-To: <20041108001901.GB477@frobozz> References: <20041107133719.F86E.JCARLSON@uci.edu> <Pine.LNX.4.10.10411071531210.25017-100000@sumeru.stanford.EDU> <20041108001901.GB477@frobozz> Message-ID: <418F0921.5040905@v.loewis.de> Andrew Bennetts wrote: > One such list is here: > http://mail.zope.org/pipermail/zope3-dev/2002-October/003235.html I fail to see any of these as problematic: 1. invokes readable/writable in each round, thereby not preserving state; presumably ok for select and poll, but bad (wasting performance for kqueue). I can't see this as a problem: asyncore does use select/poll, not kqueue. In doing so, it first processes all ready file descriptors before going to the next round, so in the next round, it needs to check all dispatchers again. There seems to be an implicit assertion that anything that uses select/poll must be evil, and the only true API is kqueue. I can't claim to understand the rationale for introducing kqueue in the first place, but if it is to improve performance, then I expect that any performance gained in kqueue over select goes away by using Python (e.g. adding an indirection in dispatching is probably more expensive than what kqueue would have saved). 2. ties together protocol and transport. So what? I don't want to use SMTP over UDP, or X.25. TCP, possibly together with TLS, is just fine. 3. tied to sockets. Again: so what? I can't follow the assertion that you cannot use pyOpenSSL because of that, but then, I haven't used pyOpenSSL. The only possible interpretation is that pyOpenSSL does not expose "raw" (pollable) socket objects, which would really sound like a limitation in pyOpenSSL, not in asyncore. a. Cannot use NT I/O completion ports. Again, what's wrong with select? Performance? I'd really like to see a Python application where the speed gain from IOCP is significant compared to using select. The limit of 64 sockets is serious, but Python bumps it to 512. b. cannot do the the TCP transfer code in the C networking core. I don't really understand that point. What is the TCP transfer code? Regards, Martin From ncoghlan at iinet.net.au Mon Nov 8 10:51:14 2004 From: ncoghlan at iinet.net.au (Nick Coghlan) Date: Mon Nov 8 10:51:18 2004 Subject: [Python-Dev] Re: PEP 336, "Make None Callable", by Andrew McClelland In-Reply-To: <418E6782.6060004@v.loewis.de> References: <E1CPOSu-0005MB-RF@sc8-pr-cvs1.sourceforge.net> <41899E5D.2080303@interlink.com.au> <864d3709041107090070849085@mail.gmail.com> <418E6782.6060004@v.loewis.de> Message-ID: <418F4192.7020306@iinet.net.au> Martin v. L?wis wrote: > The simplest solution to achieve the rationale of the PEP would be > to add a function operator.nop. Of course, writing > > def nop(*args):pass > > isn't that more difficult than writing > > from operator import nop Although the def version above has problems with keyword arguments. Still, I'd be +0 on def nop(*args, **kwds): pass in the operator module, in the spirit of TOOWTDI. And I meant to add a resounding 'Aye!' to Tim's comment about the debugging value of None not being callable - that's an obvious indicator of 'I screwed up', whereas silently returning None as proposed in the PEP may trigger the error report an arbitrary distance from the original mistake. Cheers, Nick. -- Nick Coghlan | Brisbane, Australia Email: ncoghlan@email.com | Mobile: +61 409 573 268 From jlg at dds.nl Mon Nov 8 11:07:38 2004 From: jlg at dds.nl (Johannes Gijsbers) Date: Mon Nov 8 11:05:08 2004 Subject: [Python-Dev] Bug Day in review Message-ID: <20041108100738.GA5840@surfboy> Well, Python Bug Day 4 was held yesterday. 12 patches and 10 bugs were closed. Also, there are some bugs that are almost closable on http://python.org/moin/PythonBugDayStatus. These need to be reviewed by a committer. Things I noticed: * One one hand, no regressions were introduced, as far as I can tell. On the other hand, it's pretty hard to fix a lot of bugs if you're trying to keep regressions to a minimum. * Most people just don't know where to start with fixing the bugs. We had a list of 'easy-to-fix' bugs on the second bug day, and it worked out pretty well. We should probably prepare one for the next one as well. * The Python bug list may seem pretty large, but it isn't as bad it looks. Most of the bugs are either a corner case or a design limitation. Both types are hard to fix; there are very few glaringly obvious, easy-to-fix bugs left. * I'm still too much of a newbie developer to run a bug day completely by myself. I can help out people with the process of fixing bugs, but I can't simultaneously learn about new parts of the Python source. Conclusions for the next bug day: * I'm willing to organize another bug day, but I'm not going to do it alone. A combination of a more experienced developer (for the hard technical decisions/discussions) and me (for introducing new developers to the process) would probably work out better. * There should be a way to mark bugs as "easy-to-fix". I'll try to work in this into Roundup somehow, but until we migrate to that, I propose that if anyone (committer or otherwise) sees a bug that seems easy to fix for a new developer, (s)he should add it to the PythonBugDayStatus page. * Right before a release probably isn't the best time to hold a bug day. Cheers, Johannes From ncoghlan at iinet.net.au Mon Nov 8 13:40:53 2004 From: ncoghlan at iinet.net.au (Nick Coghlan) Date: Mon Nov 8 13:40:59 2004 Subject: [Python-Dev] Pre-PEP: Executing modules inside packages with '-m' Message-ID: <418F6955.5000509@iinet.net.au> A little later than planned, but I have a draft of the PEP to extend the -m switch to cover modules inside packages. I will note that the SF reference implementation linked to by the PEP is marginally out of date - under certain circumstances (e.g. "python -mpychecker.checker") the current patch on SF can overwrite the C-level argv[0] which I've realised may not be a great idea given the existence of the Py_GetArgcArgv API (the PEP text describes the planned workaround). Regards, Nick. -- Nick Coghlan | Brisbane, Australia Email: ncoghlan@email.com | Mobile: +61 409 573 268 -------------- next part -------------- PEP: XXX Title: Executing modules inside packages with '-m' Version: $Revision: 1.4 $ Last-Modified: $Date: 2003/09/22 04:51:50 $ Author: Nick Coghlan <ncoghlan@email.com>, Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 16-Oct-2004 Python-Version: 2.5 Post-History: 16-Oct-2004 Abstract ======== This PEP defines semantics for executing modules inside packages as scripts with the ``-m`` command line switch. The proposed semantics are that the containing package be imported prior to execution of the script. Rationale ========= Python 2.4 adds the command line switch ``-m`` to allow modules to be located using the Python module namespace for execution as scripts. The motivating examples were standard library modules such as ``pdb`` and ``profile``. A number of users and developers have requested extension of the feature to also support running modules located inside packages. One example provided is pychecker's ``pychecker.checker`` module. This capability was left out of the Python 2.4 implementation because the appropriate semantics are not entirely clear. The opinion on python-dev was that it was better to postpone the extension to Python 2.5, and go through the PEP process to help make sure we got it right. Scope of this proposal ========================== In Python 2.4, a module located using ``-m`` is executed just as if its filename had been provided on the command line. The goal of this PEP is to get as close as possible to making that statement also hold true for modules inside packages. Prior discussions suggest it should be noted that this PEP is **not** about any of the following: - changing the idiom for making Python modules also useful as scripts. - lifting the restriction of ``-m`` to modules of type PY_SOURCE or PY_COMPILED (i.e. ``.py``, ``.pyc``, ``.pyo``,``.pyw``). - addressing the problem of ``-m`` not understanding zip imports or Python's sys.metapath. The issues listed above are considered orthogonal to the specific feature addressed by this PEP. Current Behaviour ================= Before describing the new semantics, it's worth covering the existing semantics for Python 2.4 (as they are currently defined only by the source code). When ``-m`` is used on the command line, it immediately terminates the option list (like ``-c``). The argument is interpreted as the name of a top-level Python module (i.e. one which can be found on ``sys.path``). If the module is found, and is of type ``PY_SOURCE`` or ``PY_COMPILED``, then the command line is effectively reinterpreted from ``python <options> -m <module> <args>`` to ``python <options> <filename> <args>``. This includes setting ``sys.argv[0]`` correctly (some scripts rely on this - Python's own ``regrtest.py`` is one example). If the module is not found, or is not of the correct type, an error is printed. Proposed Semantics ================== The semantics proposed are fairly simple: if ``-m`` is used to execute a module inside a package as a script, then the containing package is imported before executing the module in accordance with the semantics for a top-level module. This is necessary due to the way Python's import machinery locates modules inside packages. A package may modify its own __path__ variable during initialisation. Accordingly, the only way for Python to reliably locate the module is by importing the containing package and inspecting its __path__ variable. Note that the package is *not* imported into the ``__main__`` module's namespace. The effects of these semantics that will be visible to the executed module are: - the containing package will be in sys.modules - any external effects of the package initialisation (e.g. installed import hooks) Reference Implementation ======================== A reference implementation is available on SourceForge [1]_. In this implementation , if the ``-m`` switch fails to locate the requested module at the top level, it effectively reinterprets the command from ``python -m <script>`` to ``python -m execmodule <script>``. (There is one caveat: when reinterpreted in this way, ``sys.argv[0]`` may not actually contain the filename of ``execmodule``. This only affects ``execmodule`` itself, not the requested module). ``execmodule`` is a proposed standard library module that contains a single function (also called ``execmodule``). When invoked as a script, this module finds and executes the module supplied as the first argument. It adjusts ``sys.argv`` by deleting ``sys.argv[0]`` and replacing the new ``sys.argv[0]`` with the module's filename instead of its Python name. The function ``execmodule`` is like ``execfile``, but uses the Python module namespace to locate the script instead of the filesystem. It has an additional optional argument ``set_argv0`` which causes the filename of the located module to be written to ``sys.argv[0]`` before the module is executed. A hybrid C/Python implementation is used as the Python module is much more flexible and extensible than the equivalent C code would be. It also allows the ``execmodule`` function to be made available. The Python code for ``execmodule`` has also been posted as a cookbook recipe for Python 2.4 [2]_. Open Issues =========== - choosing a name for the standard library module containing ``execmodule``. The reference implementation uses ``execmodule``. An alternative name proposed on python-dev is ``runpy``. Alternatives ============ The main alternative implementation considered ignored packages' __path__ variables, and looked only in the main package directory. However, this approach did not meet the main goal of the ``-m`` switch - to allow the full Python namespace to be used to locate modules for execution. References ========== .. [1] Native ``-m`` execmodule support (http://www.python.org/peps/pep-0009.html) .. [2] execmodule Python Cookbook Recipe (http://www.python.org/peps/pep-0001.html) Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 End: From ncoghlan at iinet.net.au Mon Nov 8 14:15:27 2004 From: ncoghlan at iinet.net.au (Nick Coghlan) Date: Mon Nov 8 14:15:35 2004 Subject: [Python-Dev] Pre-PEP: Executing modules inside packages with '-m' In-Reply-To: <418F6955.5000509@iinet.net.au> References: <418F6955.5000509@iinet.net.au> Message-ID: <418F716F.1030304@iinet.net.au> Oops - forgot to update the links properly. Correct links are: [1] Native ``-m`` execmodule support http://sourceforge.net/tracker/?func=detail&aid=1043356&group_id=5470&atid=305470 [2] execmodule Python Cookbook Recipe http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/307772 Cheers, Nick. -- Nick Coghlan | Brisbane, Australia Email: ncoghlan@email.com | Mobile: +61 409 573 268 From aleaxit at yahoo.com Mon Nov 8 14:52:09 2004 From: aleaxit at yahoo.com (Alex Martelli) Date: Mon Nov 8 14:52:23 2004 Subject: [Python-Dev] Synchronous and Asynchronous servers in the standard library In-Reply-To: <418F0921.5040905@v.loewis.de> References: <20041107133719.F86E.JCARLSON@uci.edu> <Pine.LNX.4.10.10411071531210.25017-100000@sumeru.stanford.EDU> <20041108001901.GB477@frobozz> <418F0921.5040905@v.loewis.de> Message-ID: <6278459D-318D-11D9-ADE7-000A95EFAE9E@yahoo.com> On 2004 Nov 08, at 06:50, Martin v. L?wis wrote: > a. Cannot use NT I/O completion ports. Again, what's wrong > with select? Performance? I'd really like to see a Python > application where the speed gain from IOCP is significant Isn't it the case that, on Windows, select can only use sockets while IOCP would also enable the use of other things such as files? I remember that something like that used to be the case, but it's been a while since I did Windows, so my memory may be off or currently obsolete. But if it's accurate it seems to me that -- on Windows -- select does not allow smooth integration of non-socket sources while IOCP might. Alex From exarkun at divmod.com Mon Nov 8 16:34:34 2004 From: exarkun at divmod.com (exarkun@divmod.com) Date: Mon Nov 8 16:34:39 2004 Subject: [Python-Dev] Synchronous and Asynchronous servers in the standard library In-Reply-To: <418F0921.5040905@v.loewis.de> Message-ID: <20041108153434.14104.15431039.divmod.quotient.1818@ohm> On Mon, 08 Nov 2004 06:50:25 +0100, =?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?= <martin@v.loewis.de> wrote: >Andrew Bennetts wrote: > > One such list is here: > > http://mail.zope.org/pipermail/zope3-dev/2002-October/003235.html > > I fail to see any of these as problematic: Before continuing this discussion, it might be useful to define the goals for whatever ultimately ends up in the standard library. I believe there are a lot of differing implicit assumptions held by different posters on this topic which will make it very difficult to reach anything resembling concensus if not first resolved. Here are a few questions to get started (I'm sure there are more to be considered): What level of code re-use is desired? Should a protocol implementation be portable between different frameworks (Remind anyone of PEP 333)? Should new classes be required for different transports? eg, a class for SMTP/TCP, a class for SMTP/OpenSSL-TLS, a class for SMTP/TLSLite-TLS, a class For SMTP/TLSLite-SSL. What level of performance is desirable? Is the aim something comparable to the threading and forking based servers already in the stdlib? Is it "sky's the limit"? Should it be platform independent? Who is the target audience? Will this be something beginning network programmers should be able to pick up and use reasonably easy? Is that more important than allowing more complex software to be expressed easily? (Most of the rest of these questions are basically subsets of this one) That said... > > 1. invokes readable/writable in each round, thereby not preserving > state; presumably ok for select and poll, but bad (wasting > performance for kqueue). > I can't see this as a problem: asyncore does use select/poll, > not kqueue. In doing so, it first processes all ready file > descriptors before going to the next round, so in the next > round, it needs to check all dispatchers again. > There seems to be an implicit assertion that anything that uses > select/poll must be evil, and the only true API is kqueue. > I can't claim to understand the rationale for introducing > kqueue in the first place, but if it is to improve performance, > then I expect that any performance gained in kqueue over > select goes away by using Python (e.g. adding an indirection > in dispatching is probably more expensive than what kqueue > would have saved). KQueue's performance benefits over select() are absolutely noticable from Python. We're talking orders of magnitude stuff here. select() is fine for small servers, and poll() is even a little better, but unless you want to impose harsh, arbitrary limits on the scalability of servers (and certain kinds of clients, eg aggregators) developed in Python, these shouldn't be the only two options. Even if KQueue isn't supported directly by the standard library, the mechanism should be amenable to support of it by third parties - this means an easily overridden notification call as well as supporting assumptions from the rest of the library. Calling a function once per connection per iteration should be avoided if at all possible (and it is possible). > > 2. ties together protocol and transport. So what? I don't > want to use SMTP over UDP, or X.25. TCP, possibly together > with TLS, is just fine. Keep in mind there is no support for SSL servers in the standard library (this is still true, right? I admit I haven't been paying much attention to this lately). So right off the bat, you _can't_ tie the transport to the protocol if you want to write an SSL server. It needs to be trivially parameterizable by a third party. Even ignoring that, there are several choices for SSL implementations in Python. If it can be avoided, I don't see any reason to require a separate class for each of them for each protocol. > > 3. tied to sockets. Again: so what? I can't follow the assertion > that you cannot use pyOpenSSL because of that, but then, > I haven't used pyOpenSSL. The only possible interpretation > is that pyOpenSSL does not expose "raw" (pollable) socket > objects, which would really sound like a limitation in > pyOpenSSL, not in asyncore. The point isn't "tied to sockets" but "tied to the socket module". For a slightly less obvious example of why this is a drawback, consider that PyOpenSSL connections raise SSL.Error, not socket.error. So if you want to use PyOpenSSL, you don't just have to find and replace all of the uses of socket.socket(), but all of the uses of socket.error as well. Even worse, the behavior of SSL sockets doesn't entirely match the behavior of non-SSL sockets - select() can tell you a socket is ready for reading, and then a read can return '' because the available bytes were consumed entirely by SSL protocol-level negotiation. asyncore will interpret this to mean the socket has been closed by the peer and call handle_close()! So you really need an almost entirely different dispatcher to deal with SSL, and now we're back to the point above of having lots of different classes, one for each transport. > > a. Cannot use NT I/O completion ports. Again, what's wrong > with select? Performance? I'd really like to see a Python > application where the speed gain from IOCP is significant > compared to using select. The limit of 64 sockets is serious, > but Python bumps it to 512. Performance, yes. As Alex Martelli mentioned before me, IOCP also supports non-socket handles. stdin/stdout, for example, as well as handles to subprocesses. Again, since IOCP provides speedups comparable to KQueue when compared to select(), the speedups for large numbers of sockets is totally noticable, even from sluggish old Python ;) Of course, you'd never notice them if your server could only handle 512 connections, but maybe you'd appreciate not having that limit, too :) > > b. cannot do the the TCP transfer code in the C networking > core. I don't really understand that point. What is the > TCP transfer code? This would involve rewriting all of the socket handling, buffer handling, and even dispatch as an extension module. Buffer handling is a real win here, since implementing it in Python involves a lot of wasteful string copying. Itamar (original poster on the zope list) is now employed writing a lot of code like this for Twisted. He has very high throughput requirements, and they'd be very difficult to achieve without the ability to replace this core component with his own code. Since Twisted _is_ architected in such a way that this code is all one unit, instead of spread out among dozens, hundreds, or thousands of instances, after rewriting just a few classes, essentially any Twisted application can benefit from his speedup. Hope this clears some things up, Jp From FBatista at uniFON.com.ar Mon Nov 8 18:28:32 2004 From: FBatista at uniFON.com.ar (Batista, Facundo) Date: Mon Nov 8 18:30:19 2004 Subject: [Python-Dev] Policy about old Python versions Message-ID: <A128D751272CD411BC9200508BC2194D053C7B7C@escpl.tcp.com.ar> [Aahz] #- As soon as the new version is released, if there is no bugfix planned #- for that issue. Two versions later (i.e. 2.3 in this case) #- even if it #- looks like it ought to be fixed in 2.1 -- we haven't been #- doing bugfixes #- more than one version old. [Martin v. L?wis] #- It is (IMO) fine to close anything now that is fixed in 2.3 #- and newer. #- The proper "Resolution" is "Fixed", with a Comment "Fixed in 2.3". [Anthony Baxter] #- As for the original query, about what to do with bugs filed against #- 2.1.x - I'd suggest closing them. There's a vanishingly small chance #- that there will ever be another 2.1 or 2.2 release. A bug count from SF: Group Quantity 2.1.1 2 2.1.2 3 2.2 20 2.2.1 17 2.2.1c 1 2.2.2 35 2.2.3 24 ----- 102 Of 769 open bugs, that represents more than a 13%: I think that closing them or reassigning them to a newer version (if the problem continues) will make work with bugs a little easier. So I propose to take the job of getting into each one of these bugs and comment: """ Please, could you verify if this problem persists in Python 2.3.4 or 2.4? If yes, in which version? Can you provide a test case? If the problem is solved, from which version? Note that if you fail to answer in one month, I'll close this bug as "Won't fix". Thank you! """ If nobody answers in a month, close the bug. The worst problem that we can have here is to close a bug that still exists. What do you think of this? If you're ok, I'll do it. [Trent Mick] #- Maybe a "Known Issues in Python 2.1" PEP which refers to these bug #- numbers and then close the bugs? I don't want to make more work for #- people, though. This idea of compilate all the problems of those past versions in a PEP is good, but I can not do it: in the best case I can compile all the "once opened bugs" for those versions, but not verify that each is really a problem. A PEP with this kind of uncertainty worths it? Thanks for everything. . Facundo #- -----Mensaje original----- #- De: Trent Mick [mailto:trentm@ActiveState.com] #- Enviado el: Viernes, 05 de Noviembre de 2004 15:26 #- Para: Anthony Baxter #- CC: Aahz; Batista, Facundo; Python Dev (E-mail) #- Asunto: Re: [Python-Dev] Policy about old Python versions #- #- #- Anthony Baxter wrote: #- > As for the original query, about what to do with bugs filed against #- > 2.1.x - I'd suggest closing them. There's a vanishingly #- small chance #- > that there will ever be another 2.1 or 2.2 release. #- #- Maybe a "Known Issues in Python 2.1" PEP which refers to these bug #- numbers and then close the bugs? I don't want to make more work for #- people, though. #- #- Trent #- #- -- #- Trent Mick #- trentm@activestate.com #- -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20041108/32052fcc/attachment-0001.html From jimjjewett at yahoo.com Mon Nov 8 18:55:02 2004 From: jimjjewett at yahoo.com (Jim J Jewett) Date: Mon Nov 8 18:55:05 2004 Subject: [Python-Dev] Policy about old Python versions Message-ID: <20041108175502.48541.qmail@web50701.mail.yahoo.com> Batista, Facundo wrote: [Trent Mick] #- Maybe a "Known Issues in Python 2.1" PEP #- which refers to these bug numbers and then #- close the bugs? Batista, Facundo wrote: # the best case I can compile all the "once # opened bugs" for those versions, but not verify # that each is really a problem. Could you also indicate which ones were closed only because they were too old? In other words, if you Bug nnn was already closed for other reasons, then either don't include it on your list at all, or note in your list that it was closed (instead of just expiring). -jJ __________________________________ Do you Yahoo!? Check out the new Yahoo! Front Page. www.yahoo.com From fdrake at acm.org Mon Nov 8 18:55:35 2004 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Mon Nov 8 18:55:43 2004 Subject: [Python-Dev] Bug Day in review In-Reply-To: <20041108100738.GA5840@surfboy> References: <20041108100738.GA5840@surfboy> Message-ID: <200411081255.35450.fdrake@acm.org> On Monday 08 November 2004 05:07 am, Johannes Gijsbers wrote: > * There should be a way to mark bugs as "easy-to-fix". I'll try to work > in this into Roundup somehow, but until we migrate to that, I propose Now *this* sort of functionality is a good reason to migrate to roundup. I've generally not felt that there was any benefit to the migration, but being able to add this sort of community-support feature (assuming it actually happens) makes the cost of migration much more palatable, since there's an actual concrete benefit on the other side. This makes a lot more sense than having to wrap an external document around the SourceForge trackers. -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> From gjc at inescporto.pt Mon Nov 8 19:42:50 2004 From: gjc at inescporto.pt (Gustavo J. A. M. Carneiro) Date: Mon Nov 8 19:45:21 2004 Subject: [Python-Dev] Synchronous and Asynchronous servers in the standard library In-Reply-To: <Pine.LNX.4.10.10411070805390.25017-100000@sumeru.stanford.EDU> References: <Pine.LNX.4.10.10411070805390.25017-100000@sumeru.stanford.EDU> Message-ID: <1099939370.561.55.camel@spectrum.inescn.pt> Hello, I recently felt the need to have a main loop with timeouts and IO notifications, such as the one found in pygtk's gobject module (wrapping glib functions). However, I did not want to add a dependency of pygtk to this program. Therefore I implemented a main loop module in python. I think that both asyncore and twisted are a bit too high-level. I like the low-level event driven approach much better. It's easier to understand. That's why I wrote this. There's a unit test, and it seems to work, but this code is still recent and not very well tested. http://yang.inescn.pt/~gjc/pymainloop/ Just so you know this exists... Regards. Dom, 2004-11-07 ?s 08:11 -0800, Dennis Allison escreveu: > The question, then, is what base should be used for developing asynchonous > servers? > > On Sun, 7 Nov 2004, Jeremy Hylton wrote: > > > On Mon, 08 Nov 2004 01:32:35 +1100, Anthony Baxter > > <anthony@interlink.com.au> wrote: > > > Josiah Carlson wrote: > > > > A recent patch to offer an SMTP server (SocketServer derivative) sparked > > > > the below... > > > > > > > > Question: > > > > Does we (and by 'we' I mean those in charge of developing Python) want > > > > to offer both asynchronous (deriving from asyncore, asynchat, etc.) and > > > > synchronous versions of server software in the standard library? > > > > > > Personally, I think that encouraging anyone to develop new software on > > > top of asyncore/asynchat is a _terrible_ idea. > > > > I agree. > > > > Jeremy > > _______________________________________________ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/gjc%40inescporto.pt -- Gustavo J. A. M. Carneiro <gjc@inescporto.pt> <gustavo@users.sourceforge.net> The universe is always one step beyond logic. -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 3086 bytes Desc: not available Url : http://mail.python.org/pipermail/python-dev/attachments/20041108/6e607a5d/smime.bin From FBatista at uniFON.com.ar Mon Nov 8 19:49:46 2004 From: FBatista at uniFON.com.ar (Batista, Facundo) Date: Mon Nov 8 19:51:34 2004 Subject: [Python-Dev] Policy about old Python versions Message-ID: <A128D751272CD411BC9200508BC2194D053C7B7F@escpl.tcp.com.ar> [Jim J Jewett] #- # the best case I can compile all the "once #- # opened bugs" for those versions, but not verify #- # that each is really a problem. #- #- Could you also indicate which ones were closed #- only because they were too old? #- #- In other words, if you Bug nnn was already #- closed for other reasons, then either don't #- include it on your list at all, or note in #- your list that it was closed (instead of just #- expiring). Yes, in this old-bug-closing round I'll surely detail which bugs were closed and the reason of each closure. . Facundo -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20041108/52490e6c/attachment.htm From jlg at dds.nl Mon Nov 8 20:38:31 2004 From: jlg at dds.nl (Johannes Gijsbers) Date: Mon Nov 8 20:44:40 2004 Subject: [Python-Dev] Policy about old Python versions In-Reply-To: <A128D751272CD411BC9200508BC2194D053C7B7C@escpl.tcp.com.ar> References: <A128D751272CD411BC9200508BC2194D053C7B7C@escpl.tcp.com.ar> Message-ID: <20041108193831.GA5944@surfboy> On Mon, Nov 08, 2004 at 02:28:32PM -0300, Batista, Facundo wrote: > So I propose to take the job of getting into each one of these bugs and > comment: > > """ > Please, could you verify if this problem persists in Python 2.3.4 or 2.4? > > If yes, in which version? Can you provide a test case? > > If the problem is solved, from which version? > > Note that if you fail to answer in one month, I'll close this bug as "Won't > fix". > > Thank you! > """ Sounds good to me, with the following caveat. Before adding this comment, do a casual check whether the bug has been reproduced by another developer and/or whether you can reproduce it yourself. Don't spend too much time on it, but just check whether it's obvious the bug still exists. Wouldn't want to see bugs like http://python.org/sf/548176 or http://python.org/sf/531205 get closed because of lack of response. By the way: you know about the 'canned response' facility, right? Saves you from copy and pasting the same response over and over again. Cheers, Johannes From janssen at parc.com Mon Nov 8 21:33:32 2004 From: janssen at parc.com (Bill Janssen) Date: Mon Nov 8 21:33:46 2004 Subject: [Python-Dev] Synchronous and Asynchronous servers in the standard library In-Reply-To: Your message of "Sun, 07 Nov 2004 15:33:35 PST." <Pine.LNX.4.10.10411071531210.25017-100000@sumeru.stanford.EDU> Message-ID: <04Nov8.123335pst."58617"@synergy1.parc.xerox.com> > It might be worthwhile to list what the existing problems are with the > asyncore and asynchat libraries. There is also the problem that many > applications, Zope among them, depend upon them. And Medusa, which is also widely used (though unsupported). On the Web-SIG list last year, I brought up bringing something like Medusa (possibly even Medusa) into the standard library, to replace the widely unused HTTP servers in there already. Bill From p.f.moore at gmail.com Mon Nov 8 21:49:56 2004 From: p.f.moore at gmail.com (Paul Moore) Date: Mon Nov 8 21:50:05 2004 Subject: [Python-Dev] Policy about old Python versions In-Reply-To: <20041108193831.GA5944@surfboy> References: <A128D751272CD411BC9200508BC2194D053C7B7C@escpl.tcp.com.ar> <20041108193831.GA5944@surfboy> Message-ID: <79990c6b0411081249198ffbbc@mail.gmail.com> On Mon, 8 Nov 2004 20:38:31 +0100, Johannes Gijsbers <jlg@dds.nl> wrote: > Sounds good to me, with the following caveat. Before adding this > comment, do a casual check whether the bug has been reproduced by > another developer and/or whether you can reproduce it yourself. Don't > spend too much time on it, but just check whether it's obvious the bug > still exists. Wouldn't want to see bugs like http://python.org/sf/548176 > or http://python.org/sf/531205 get closed because of lack of response. FWIW, I tested these two bugs on 2.3 and 2.4b2, and added an appropriate comment. If there are any other such bugs, it's easy enough to do this yourself. Paul From FBatista at uniFON.com.ar Mon Nov 8 21:57:13 2004 From: FBatista at uniFON.com.ar (Batista, Facundo) Date: Mon Nov 8 21:59:07 2004 Subject: [Python-Dev] Policy about old Python versions Message-ID: <A128D751272CD411BC9200508BC2194D053C7B89@escpl.tcp.com.ar> [Johannes Gijsbers] #- Sounds good to me, with the following caveat. Before adding this #- comment, do a casual check whether the bug has been reproduced by #- another developer and/or whether you can reproduce it yourself. Don't #- spend too much time on it, but just check whether it's #- obvious the bug #- still exists. Wouldn't want to see bugs like #- http://python.org/sf/548176 #- or http://python.org/sf/531205 get closed because of lack of #- response. Ok. #- By the way: you know about the 'canned response' facility, #- right? Saves #- you from copy and pasting the same response over and over again. I know, never used it, good oportunity to learn, ;). . Facundo -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20041108/56c5223b/attachment.htm From irmen at xs4all.nl Mon Nov 8 22:25:54 2004 From: irmen at xs4all.nl (Irmen de Jong) Date: Mon Nov 8 22:25:56 2004 Subject: [Python-Dev] Bug Day in review In-Reply-To: <20041108100738.GA5840@surfboy> References: <20041108100738.GA5840@surfboy> Message-ID: <418FE462.2010708@xs4all.nl> Johannes Gijsbers wrote: > Well, Python Bug Day 4 was held yesterday. 12 patches and 10 bugs were > closed. Also, there are some bugs that are almost closable on > http://python.org/moin/PythonBugDayStatus. These need to be reviewed by > a committer. I was one of the participants, and I have to say that it was rather fun to do and contribute some patches/bug comments. And thanks to Johannes ofcourse for answering all my silly questions. [ some words about myself: I've ported Python to AmigaDOS waaay back in the python 1.5 days and so was quite familiar with the code at that time. But years have passed since then and I've not looked at the Python code anymore (besides the stdlib), instead I used Python to create Pyro and more recently, Snakelets. While not being a 'python developer', it was nice to dig around in the Python source code again on the bug day. New and yet familiar :-) ] I've only subscribed to python-dev yesterday, and will mainly be just lurking around to see what's going on. (The bug-day has sparked my interest so to speak) > * Most people just don't know where to start with fixing the bugs. We > had a list of 'easy-to-fix' bugs on the second bug day, and it worked > out pretty well. We should probably prepare one for the next one as > well. That would be very nice indeed, I had some trouble finding things that 1) I have some real knowledge of and 2) were 'easy-to-fix', like you're saying. So the 'easy-to-fix' flag for bugs could help a lot here, to make it easier for newbies like me to find stuff that we can perhaps fix (or help to fix) during a bug day. It's also easier for the organizers to find things to stick on the work list, I guess. One of the things that still puzzles me, is what you should do with old bugs that are in groups 'Python 2.1 ... 2.3', but I saw that there's already a discussion going on about this on this list :) Bye, --Irmen de Jong From tjreedy at udel.edu Mon Nov 8 22:26:35 2004 From: tjreedy at udel.edu (Terry Reedy) Date: Mon Nov 8 22:26:33 2004 Subject: [Python-Dev] Re: Policy about old Python versions References: <A128D751272CD411BC9200508BC2194D053C7B7C@escpl.tcp.com.ar><20041108193831.GA5944@surfboy> <79990c6b0411081249198ffbbc@mail.gmail.com> Message-ID: <cmooa2$thd$1@sea.gmane.org> side note: I find stuff like >>> from email.Utils import parseaddr much harder to read than the proper >>> from email.Utils import parseaddr http://python.org/sf/531205 Even worse is http://<host>:<port>/<path>?<searchpart> versus something like scheme>://<authority><path>?<query> (both quoted from http://python.org/sf/548176) SourceForge is designed (intentionally, I am sure) to accept and properly display the latter and not require chicken-scratch input. Terry J. Reedy From t-meyer at ihug.co.nz Mon Nov 8 22:52:44 2004 From: t-meyer at ihug.co.nz (Tony Meyer) Date: Mon Nov 8 22:53:31 2004 Subject: [Python-Dev] Re: Policy about old Python versions In-Reply-To: <ECBA357DDED63B4995F5C1F5CBE5B1E801521491@its-xchg4.massey.ac.nz> Message-ID: <ECBA357DDED63B4995F5C1F5CBE5B1E86C37CF@its-xchg4.massey.ac.nz> > I find stuff like > >>> from email.Utils import parseaddr > much harder to read than the proper > >>> from email.Utils import parseaddr > http://python.org/sf/531205 > > Even worse is > http://<host>:<port>/<path>?<searchpart> > versus something like > scheme>://<authority><path>?<query> >(both quoted from http://python.org/sf/548176) > > SourceForge is designed (intentionally, I am sure) > to accept and properly display the latter > and not require chicken-scratch input. Actually, this is almost certainly the result of a SF glitch (from around the start of the year, IIRC). At that time, all SF trackers (comments, summary, etc) had '<' converted to < (etc). There's no way to edit the existing comments, so the trackers are permanently stuck like that. The SF people themselves indicated that they would not be able to fix any, except manually, and that would only be done where there was a real need (I can't think of any example of that). You'll find examples of this in lots of project trackers that are suitably old. Python was probably hit harder than some, since the '>' character is probably reasonably common in comments. =Tony.Meyer From greg at electricrain.com Tue Nov 9 00:10:42 2004 From: greg at electricrain.com (Gregory P. Smith) Date: Tue Nov 9 00:10:46 2004 Subject: [Python-Dev] Synchronous and Asynchronous servers in the standard library In-Reply-To: <418F0921.5040905@v.loewis.de> References: <20041107133719.F86E.JCARLSON@uci.edu> <Pine.LNX.4.10.10411071531210.25017-100000@sumeru.stanford.EDU> <20041108001901.GB477@frobozz> <418F0921.5040905@v.loewis.de> Message-ID: <20041108231042.GA21680@zot.electricrain.com> On Mon, Nov 08, 2004 at 06:50:25AM +0100, "Martin v. L?wis" wrote: > Andrew Bennetts wrote: > >One such list is here: > > http://mail.zope.org/pipermail/zope3-dev/2002-October/003235.html > > I fail to see any of these as problematic: > > 1. invokes readable/writable in each round, thereby not preserving > state; presumably ok for select and poll, but bad (wasting > performance for kqueue). > I can't see this as a problem: asyncore does use select/poll, > not kqueue. In doing so, it first processes all ready file > descriptors before going to the next round, so in the next > round, it needs to check all dispatchers again. > There seems to be an implicit assertion that anything that uses > select/poll must be evil, and the only true API is kqueue. > I can't claim to understand the rationale for introducing > kqueue in the first place, but if it is to improve performance, > then I expect that any performance gained in kqueue over > select goes away by using Python (e.g. adding an indirection > in dispatching is probably more expensive than what kqueue > would have saved). The complaint about the readable and writable methods is partially the overhead of doing a method call for each file descriptor every time thru the loop. Its inefficient. Itamar mentions that both select() and poll() effectively maintain a reusable data structure that could be maintained directly by the file descriptor event handlers rather than expensively querying them all each time thru the loop. In any given iteration thru the loop its typical for only a small fraction of the file desciptors to change readable/writable state so why not have them manage their flag directly. (analogy: a push rather than a pull of the readable/writable state to the event loop) -g From Andrew.MacKeith at ABAQUS.com Mon Nov 8 21:46:04 2004 From: Andrew.MacKeith at ABAQUS.com (Andrew MacKeith) Date: Tue Nov 9 06:32:24 2004 Subject: [Python-Dev] Synchronous and Asynchronous servers in the standard library References: <04Nov8.123335pst."58617"@synergy1.parc.xerox.com> Message-ID: <418FDB0C.9090305@ABAQUS.com> Bill Janssen wrote: >>It might be worthwhile to list what the existing problems are with the >>asyncore and asynchat libraries. There is also the problem that many >>applications, Zope among them, depend upon them. > > > And Medusa, which is also widely used (though unsupported). On the > Web-SIG list last year, I brought up bringing something like Medusa > (possibly even Medusa) into the standard library, to replace the > widely unused HTTP servers in there already. Unused? I use them when I need a simple server. Andrew > > Bill > _______________________________________________ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/mackeith%40acm.org > From martin at v.loewis.de Tue Nov 9 06:49:06 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue Nov 9 06:49:08 2004 Subject: [Python-Dev] Synchronous and Asynchronous servers in the standard library In-Reply-To: <6278459D-318D-11D9-ADE7-000A95EFAE9E@yahoo.com> References: <20041107133719.F86E.JCARLSON@uci.edu> <Pine.LNX.4.10.10411071531210.25017-100000@sumeru.stanford.EDU> <20041108001901.GB477@frobozz> <418F0921.5040905@v.loewis.de> <6278459D-318D-11D9-ADE7-000A95EFAE9E@yahoo.com> Message-ID: <41905A52.2080507@v.loewis.de> Alex Martelli wrote: > But if it's accurate it seems to me that -- on Windows -- > select does not allow smooth integration of non-socket sources while > IOCP might. I believe this is right. I don't see this as a problem, though - in the typical asyncore application, what non-socket sources would you have? Regards, Martin From martin at v.loewis.de Tue Nov 9 07:07:56 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue Nov 9 07:07:56 2004 Subject: [Python-Dev] Synchronous and Asynchronous servers in the standard library In-Reply-To: <20041108153434.14104.15431039.divmod.quotient.1818@ohm> References: <20041108153434.14104.15431039.divmod.quotient.1818@ohm> Message-ID: <41905EBC.2060107@v.loewis.de> exarkun@divmod.com wrote: > KQueue's performance benefits over select() are absolutely noticable > from Python. We're talking orders of magnitude stuff here. select() > is fine for small servers Can you quantify this a bit? How can I tell whether my server is small, and how do I notice a the difference? > Keep in mind there is no support for SSL servers in the standard > library (this is still true, right? True. > If it can be avoided, I don't see any reason to require a separate > class for each of them for each protocol. I'm not sure it can be avoided. But then, I have not implemented any SSL server in Python. I would think that - in SMTP, you *must* reuse the socket, since the SSL modes starts only after the STARTTLS command - in HTTP, you need to be SSL-aware, in order to pass the server key to the SSL library, and in order to validate the client certificate (if the client choses to send one) IOW, whether or not SSL is used causes changes beyond the transport, and specific to the protocol and application logic. > asyncore will interpret this to mean > the socket has been closed by the peer and call handle_close()! I see. Regards, Martin From pje at telecommunity.com Tue Nov 9 07:09:34 2004 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue Nov 9 07:09:22 2004 Subject: [Python-Dev] Synchronous and Asynchronous servers in the standard library In-Reply-To: <41905A52.2080507@v.loewis.de> References: <6278459D-318D-11D9-ADE7-000A95EFAE9E@yahoo.com> <20041107133719.F86E.JCARLSON@uci.edu> <Pine.LNX.4.10.10411071531210.25017-100000@sumeru.stanford.EDU> <20041108001901.GB477@frobozz> <418F0921.5040905@v.loewis.de> <6278459D-318D-11D9-ADE7-000A95EFAE9E@yahoo.com> Message-ID: <5.1.1.6.0.20041109010737.02cd1710@mail.telecommunity.com> At 06:49 AM 11/9/04 +0100, Martin v. L?wis wrote: >Alex Martelli wrote: >>But if it's accurate it seems to me that -- on Windows -- select does not >>allow smooth integration of non-socket sources while IOCP might. > >I believe this is right. I don't see this as a problem, though - in the >typical asyncore application, what non-socket sources would you have? Subprocess pipes would be one example, although I suppose you could rule that out as "atypical", but then that's just saying that anything you can't already do with it isn't typical use for it. I have, however, written event-driven apps that talk to subprocess pipes and sockets at the same time. Just not using asyncore. :) From exarkun at divmod.com Tue Nov 9 08:12:30 2004 From: exarkun at divmod.com (Jp Calderone) Date: Tue Nov 9 08:12:33 2004 Subject: [Python-Dev] Synchronous and Asynchronous servers in the standard library In-Reply-To: <41905EBC.2060107@v.loewis.de> Message-ID: <20041109071230.20649.1802519922.divmod.quotient.445@ohm> On Tue, 09 Nov 2004 07:07:56 +0100, Martin v. Löwis <martin@v.loewis.de> wrote: >exarkun@divmod.com wrote: > > KQueue's performance benefits over select() are absolutely noticable > > from Python. We're talking orders of magnitude stuff here. select() > > is fine for small servers > > Can you quantify this a bit? How can I tell whether my server is small, > and how do I notice a the difference? > See http://www.kegel.com/dkftpbench/Poller_bench.html#setup.solaris for some specific numbers, as well as a link to a simple benchmark tool you can try out (if you have a BSD machine). None of these speak to Python overhead, but even assuming a huge number, like 30ms per event notification (which we choose to apply only to KQueue and not poll, because we're giving poll the benefit of the doubt), KQueue is already a win at 100 sockets. Most important here is that KQueue (and related mechanisms - IOCP on Windows, and AIO on Linux (if it supported sockets ;)) scale with the number of active sockets, rather than the total number of sockets, as the other schemes do. A KQueue server with 1000 quiescent clients and 1 active client is paying the same cost as a KQueue server with 1 active client; a poll server is paying between 500x and 1000x as much. > > Keep in mind there is no support for SSL servers in the standard > > library (this is still true, right? > > True. > > > If it can be avoided, I don't see any reason to require a separate > > class for each of them for each protocol. > > I'm not sure it can be avoided. But then, I have not implemented any SSL > server in Python. I would think that > - in SMTP, you *must* reuse the socket, since the SSL modes starts > only after the STARTTLS command SMTP/SSL actually exists; I don't believe it is terribly widely used, but it is an alternative to the STARTTLS ESMTP protocol action. Let's ignore that for the moment though. The case of switching between a plain TCP connection and an encrypted SSL connection is an interesting one. Say we have an SMTP/TCP server class, SMTP_TCP. It inherits a buffering, error handling, etc, write() method from some helpful base class. This version of write() handles socket.errors and deals with short send()s and so forth. How can we write SMTP_TLS? If it is a subclass of SMTP_TCP, then it inherits all the wrong socket behavior, because eventually it will need to deal with SSL send()s and recv()s. We could create a mixin that provides an SSL-aware implementation of the write() method, but this will be wrong for the part of the connection that occurs before STARTTLS is issued. What options are there? There are a few unspeakables (runtime class creation, runtime inheritence graph manipulation, etc), and just a couple that are halfway reasonable: override all of the socket-related methods, add a conditional that calls TCP implementations before STARTTLS and SSL implementations afterwards; or add bound methods as attributes to the instance when STARTTLS arrives, essentially overriding the TCP implementations with the SSL implementations. As for the first of these, it's a lot of code duplication, especially when you consider how many protocols have the concept of STARTTLS and will need to do the same thing. A mixin could help reduce this, although the fragile dependence of this solution on the exact shape of the inheritence tree is something I've had to deal with and would not look forward to encountering again. At best, it will be obscure. For the latter, well, that sounds like a lot of extra work to just duplicate what you'd get by dispatching calls to a 2nd object, and then replacing that object when STARTTLS arrives. Kind of like... def cmd_STARTTLS(self): self.transport.write('220 Begin TLS negotiation now\r\n') self.transport = SSLConnection(self.transport) This strikes me as downright elegent, and would continue to do so even were I to forget what it is being compared to ;) > - in HTTP, you need to be SSL-aware, in order to pass the server > key to the SSL library, and in order to validate the client > certificate (if the client choses to send one) > IOW, whether or not SSL is used causes changes beyond the transport, > and specific to the protocol and application logic. These can all be handled with relative ease. Since I was so long winded above, I'll try and keep this short: def cmd_STARTTLS(self): if self.transport.canStartSSL(): self.transport.write('220 Begin TLS negotiation now\r\n') self.transport = SSLConnection(self.transport) else: self.transport.write('454 TLS not available\r\n') The exact spelling of "self.transport.canStartSSL()" can go in any of a number of ways, but that is a different discussion. ;) Point being, graceful degradation of functionality is entirely possible, and arguably easier (or at least more logically structured) than the class-based solutions, which would simply present import errors in the absence of necessary SSL support, or would require the equivalent of the "self.transport.canStartSSL()" check somewhere out in the application code, rather than nicely encapsulated within the implementation of STARTTLS itself. Another way to put all of this is that by separating the protocol from the transport, the library is required to provide P + T classes, where P is the number of supported protocols and T is the number of supported transports. By statically defining all permutations of the two, the library is required to provide P * T classes. Aside from the extra effort required, many users won't be interested in anything near the full permutation space in any single application, and so will be just as happy to create the pairs themselves, as necessary. Jp From lunz at falooley.org Tue Nov 9 16:18:58 2004 From: lunz at falooley.org (Jason Lunz) Date: Tue Nov 9 16:19:05 2004 Subject: [Python-Dev] Re: Synchronous and Asynchronous servers in the standard library References: <6278459D-318D-11D9-ADE7-000A95EFAE9E@yahoo.com> <20041107133719.F86E.JCARLSON@uci.edu> <Pine.LNX.4.10.10411071531210.25017-100000@sumeru.stanford.EDU> <20041108001901.GB477@frobozz> <418F0921.5040905@v.loewis.de> <6278459D-318D-11D9-ADE7-000A95EFAE9E@yahoo.com> <41905A52.2080507@v.loewis.de> <5.1.1.6.0.20041109010737.02cd1710@mail.telecommunity.com> Message-ID: <cmqn52$j1l$1@sea.gmane.org> pje@telecommunity.com said: > Subprocess pipes would be one example, although I suppose you could rule > that out as "atypical", but then that's just saying that anything you can't > already do with it isn't typical use for it. I have, however, written > event-driven apps that talk to subprocess pipes and sockets at the same > time. Just not using asyncore. :) I need to combine subprocess pipes and sockets in an async loop too. I wouldn't call it atypical at all. Jason From db3l at fitlinxx.com Wed Nov 10 00:34:34 2004 From: db3l at fitlinxx.com (David Bolen) Date: Wed Nov 10 00:34:41 2004 Subject: [Python-Dev] Re: Synchronous and Asynchronous servers in the standard library References: <20041107133719.F86E.JCARLSON@uci.edu> <Pine.LNX.4.10.10411071531210.25017-100000@sumeru.stanford.EDU> <20041108001901.GB477@frobozz> <418F0921.5040905@v.loewis.de> <6278459D-318D-11D9-ADE7-000A95EFAE9E@yahoo.com> <41905A52.2080507@v.loewis.de> Message-ID: <u4qjya96d.fsf@fitlinxx.com> "Martin v. L?wis" <martin@v.loewis.de> writes: > Alex Martelli wrote: > > But if it's accurate it seems to me that -- on Windows -- > > select does not allow smooth integration of non-socket sources while > > IOCP might. > > I believe this is right. I don't see this as a problem, though - in the > typical asyncore application, what non-socket sources would you have? On Windows, I miss the ability to mix in serial port I/O the most, since many of my servers are translators between the network and serial devices. -- David From raymond.hettinger at verizon.net Wed Nov 10 06:33:25 2004 From: raymond.hettinger at verizon.net (Raymond Hettinger) Date: Wed Nov 10 06:35:45 2004 Subject: [Python-Dev] Deprecate PyRange_New() Message-ID: <000001c4c6e6$cda46fa0$6628a044@oemcomputer> A while back, Tim added a comment that PyRange_New() should be deprecated in-part because it is filled with problems and is not used anywhere in the core. Doesn't anyone mind if I mark it in the C-API docs as deprecated so it can be phased out later? Raymond Hettinger From martin at v.loewis.de Wed Nov 10 08:36:22 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed Nov 10 08:36:23 2004 Subject: [Python-Dev] Synchronous and Asynchronous servers in the standard library In-Reply-To: <20041109071230.20649.1802519922.divmod.quotient.445@ohm> References: <20041109071230.20649.1802519922.divmod.quotient.445@ohm> Message-ID: <4191C4F6.2040604@v.loewis.de> Jp Calderone wrote: > See http://www.kegel.com/dkftpbench/Poller_bench.html#setup.solaris > for some specific numbers, as well as a link to a simple benchmark > tool you can try out (if you have a BSD machine). None of these > speak to Python overhead, but even assuming a huge number, like 30ms > per event notification (which we choose to apply only to KQueue and > not poll, because we're giving poll the benefit of the doubt), KQueue > is already a win at 100 sockets. Most important here is that KQueue > (and related mechanisms - IOCP on Windows, and AIO on Linux (if it > supported sockets ;)) scale with the number of active sockets, rather > than the total number of sockets, as the other schemes do. A KQueue > server with 1000 quiescent clients and 1 active client is paying the > same cost as a KQueue server with 1 active client; a poll server is > paying between 500x and 1000x as much. Again, my question is: so what? If I have 1000 idle clients, and one active client, it appears that poll will need 1ms per call (1184 ?s on a 650 MHz dual Pentium). This is followed by the actual processing, which will take *much* longer (depending on the nature of processing, of course). A http server, for example, would need to parse the request, locate the file, and start sending it back. This might take 10ms and more. Perhaps people really build Python servers which have simultaneously 30000 sockets open; I don't know. > Say we have an SMTP/TCP server class, SMTP_TCP. It inherits a > buffering, error handling, etc, write() method from some helpful base > class. This version of write() handles socket.errors and deals with > short send()s and so forth. How can we write SMTP_TLS? If it is a > subclass of SMTP_TCP, then it inherits all the wrong socket behavior, > because eventually it will need to deal with SSL send()s and recv()s. I fail to see the problem. If the file object is replaced/rewrapped into a SSL object, the base class can continue to use read/write/send/recv - if only the SSL object supported the same operations (which it really should if switching from plain sockets to SSL sockets is a common task) (and you seem to be suggesting that as a good strategy). My point here is that you just cannot use the very same class for SMTP, and SMTP+STARTTLS - you indicated that a subclass must be created. This was my original point: you cannot instantiate the baseclass for use with TLS, as the baseclass does not implement the cmd_STARTTLS method. > These can all be handled with relative ease. Since I was so long > winded above, I'll try and keep this short: > > def cmd_STARTTLS(self): > if self.transport.canStartSSL(): > self.transport.write('220 Begin TLS negotiation now\r\n') > self.transport = SSLConnection(self.transport) > else: > self.transport.write('454 TLS not available\r\n') > > The exact spelling of "self.transport.canStartSSL()" can go in any of > a number of ways, but that is a different discussion. ;) This cannot work, atleast not for the problem I was discussing: In https, there is no starttls command to be implemented in a subclass. Instead, the TLS negotiation occurs at connection establishment, or transparently at any point during the conversation. The base class knows nothing about certificates, and client authentification. So again, we need another class that deals with these aspects of the protocol. Those aspects are *tied* to the transport - so the resulting class will be transport-specific. > Another way to put all of this is that by separating the protocol > from the transport, the library is required to provide P + T classes, > where P is the number of supported protocols and T is the number of > supported transports. By statically defining all permutations of the > two, the library is required to provide P * T classes. Aside from > the extra effort required, many users won't be interested in anything > near the full permutation space in any single application, and so > will be just as happy to create the pairs themselves, as necessary. Who has required the library to provide P*T classes? That was a very foolish design decision, as the users only need a small selection of combinations. As a library designer, I would have created those pairs that my users actually need. Regards, Martin From martin at v.loewis.de Wed Nov 10 08:37:28 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed Nov 10 08:37:27 2004 Subject: [Python-Dev] Re: Synchronous and Asynchronous servers in the standard library In-Reply-To: <u4qjya96d.fsf@fitlinxx.com> References: <20041107133719.F86E.JCARLSON@uci.edu> <Pine.LNX.4.10.10411071531210.25017-100000@sumeru.stanford.EDU> <20041108001901.GB477@frobozz> <418F0921.5040905@v.loewis.de> <6278459D-318D-11D9-ADE7-000A95EFAE9E@yahoo.com> <41905A52.2080507@v.loewis.de> <u4qjya96d.fsf@fitlinxx.com> Message-ID: <4191C538.4050209@v.loewis.de> David Bolen wrote: > On Windows, I miss the ability to mix in serial port I/O the most, > since many of my servers are translators between the network and > serial devices. Do you know whether IOCP could be used with the serial ports? Regards, Martin From walter at livinglogic.de Wed Nov 10 12:38:26 2004 From: walter at livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=) Date: Wed Nov 10 12:38:29 2004 Subject: [Python-Dev] Code coverage tool updated In-Reply-To: <1f7befae0411051406711618cf@mail.gmail.com> References: <41864572.4000806@livinglogic.de> <16774.45652.835103.327656@montanaro.dyndns.org> <4187765D.3000705@livinglogic.de> <4187CA1C.9050508@livinglogic.de> <1f7befae04110210537c35e514@mail.gmail.com> <418A269B.6010207@livinglogic.de> <1f7befae0411051406711618cf@mail.gmail.com> Message-ID: <4191FDB2.2000002@livinglogic.de> Tim Peters wrote: > [Walter D?rwald] > >>I guess it's not worth it to try to fix doctest/trace to >>provide sourcecode for doctest code, but IMHO trace should >>be able to survive a doctest. > > Sure! The assert also triggers for "@test" on my box, BTW -- and also > did in 2.3.4. This seems to be fixed now. > [...] >>The trace call looked like this: >>./python ../../../trace.py --count --summary --missing Lib/test/regrtest.py >>with ../../../trace.py being the trace.py from current >>CVS with the assert statement removed. >> >>So am I doing something wrong or is trace.py broken? > > Sorry, I don't know. trace.py is like voting to me -- I'm highly in > favor of it, but never have time for it <0.5 wink>. I only dropped in > here to explain the source of the synthesized doctest "file name". > > FWIW, doing what you did with current CVS Python on Windows, I get > results similar to yours: only 30-some modules reported at the end. > > Under my 2.3.4 installation instead, the same thing reports on over > 300 modules, so best guess is that trace.py is indeed suffering > breakage introduced in 2.4. No idea about specifics, though. Using regrtest.py's tracing option instead of trace.py gives better results (see http://styx.livinglogic.de/~walter/brokentrace2.txt): 656 covered modules (including a strange /tmp/tmp6ccfd4/t5/string.py) There seems to be something wrong with the way trace.py starts the script. BTW, I'd like to have an option the specify the coverdir in regrtest.py, as now finding the coverage file when you have the name of the Python file is more complicated. But I guess this has to wait until 2.4 is out the door. Bye, Walter D?rwald From aahz at pythoncraft.com Wed Nov 10 15:38:36 2004 From: aahz at pythoncraft.com (Aahz) Date: Wed Nov 10 15:38:39 2004 Subject: [Python-Dev] Deprecate PyRange_New() In-Reply-To: <000001c4c6e6$cda46fa0$6628a044@oemcomputer> References: <000001c4c6e6$cda46fa0$6628a044@oemcomputer> Message-ID: <20041110143836.GB28182@panix.com> On Wed, Nov 10, 2004, Raymond Hettinger wrote: > > A while back, Tim added a comment that PyRange_New() should be > deprecated in-part because it is filled with problems and is not used > anywhere in the core. > > Doesn't anyone mind if I mark it in the C-API docs as deprecated so it > can be phased out later? +1 +1 on emitting a compiler warning in 2.5 (a bit late for 2.4, I think). -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ WiFi is the SCSI of the 21st Century -- there are fundamental technical reasons for sacrificing a goat. (with no apologies to John Woods) From aahz at pythoncraft.com Wed Nov 10 15:49:00 2004 From: aahz at pythoncraft.com (Aahz) Date: Wed Nov 10 15:49:01 2004 Subject: [Python-Dev] What is docutils? In-Reply-To: <418CA509.2050103@v.loewis.de> References: <5.1.1.6.0.20041105101943.02b94c80@mail.telecommunity.com> <41896727.7030206@v.loewis.de> <4188BE21.6040205@interlink.com.au> <41896727.7030206@v.loewis.de> <5.1.1.6.0.20041105101943.02b94c80@mail.telecommunity.com> <5.1.1.6.0.20041105145001.02dee410@mail.telecommunity.com> <418CA509.2050103@v.loewis.de> Message-ID: <20041110144859.GC28182@panix.com> [Irregular side-note: could people please trim the Cc: line when responding to python-dev posts?] On Sat, Nov 06, 2004, "Martin v. L?wis" wrote: > > I'm still uncertain what the point behind the docutils initiative is. It's very simple: to provide roughly 80% of the power of DocBook in a human-readable text format. Everything else comes from the uses people desire to put that capability to. An eventual goal is to make reST the standard markup for Python comments, but it's actually not advanced enough for that purpose yet. (That's my opinion, but of course docutils itself is mostly documented using reST.) People are using reST for many different other purposes, though: writing books, wiki markup, web pages (see the recent www.python.org partial changeover), and so on. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ WiFi is the SCSI of the 21st Century -- there are fundamental technical reasons for sacrificing a goat. (with no apologies to John Woods) From phd at mail2.phd.pp.ru Wed Nov 10 16:32:31 2004 From: phd at mail2.phd.pp.ru (Oleg Broytmann) Date: Wed Nov 10 16:32:34 2004 Subject: [Python-Dev] Reply, Group-Reply, List-Reply (was: What is docutils?) In-Reply-To: <20041110144859.GC28182@panix.com> References: <5.1.1.6.0.20041105101943.02b94c80@mail.telecommunity.com> <41896727.7030206@v.loewis.de> <4188BE21.6040205@interlink.com.au> <41896727.7030206@v.loewis.de> <5.1.1.6.0.20041105101943.02b94c80@mail.telecommunity.com> <5.1.1.6.0.20041105145001.02dee410@mail.telecommunity.com> <418CA509.2050103@v.loewis.de> <20041110144859.GC28182@panix.com> Message-ID: <20041110153230.GC22306@phd.pp.ru> On Wed, Nov 10, 2004 at 09:49:00AM -0500, Aahz wrote: > [Irregular side-note: could people please trim the Cc: line when > responding to python-dev posts?] python-dev@ is almost always exactly in the Cc header. That because people use Group-Reply (Reply-to-All) command in their mailers. List-Reply is very unusual command in MUAs. Oleg. -- Oleg Broytmann http://phd.pp.ru/ phd@phd.pp.ru Programmers don't die, they just GOSUB without RETURN. From barry at python.org Wed Nov 10 17:17:09 2004 From: barry at python.org (Barry Warsaw) Date: Wed Nov 10 17:17:19 2004 Subject: [Python-Dev] Reply, Group-Reply, List-Reply (was: What is docutils?) In-Reply-To: <20041110153230.GC22306@phd.pp.ru> References: <5.1.1.6.0.20041105101943.02b94c80@mail.telecommunity.com> <41896727.7030206@v.loewis.de> <4188BE21.6040205@interlink.com.au> <41896727.7030206@v.loewis.de> <5.1.1.6.0.20041105101943.02b94c80@mail.telecommunity.com> <5.1.1.6.0.20041105145001.02dee410@mail.telecommunity.com> <418CA509.2050103@v.loewis.de> <20041110144859.GC28182@panix.com> <20041110153230.GC22306@phd.pp.ru> Message-ID: <1100103429.20402.15.camel@geddy.wooz.org> On Wed, 2004-11-10 at 10:32, Oleg Broytmann wrote: > On Wed, Nov 10, 2004 at 09:49:00AM -0500, Aahz wrote: > > [Irregular side-note: could people please trim the Cc: line when > > responding to python-dev posts?] > > python-dev@ is almost always exactly in the Cc header. That because > people use Group-Reply (Reply-to-All) command in their mailers. > List-Reply is very unusual command in MUAs. The best you can do is go to your Mailman option's page for python-dev and select "Yes" for "Avoid duplicate copies of messages?". The effect of this is to have Mailman suppress sending you a list copy if your email address is explicitly named in the recipient headers. It's not ideal, but it's all Mailman can do. Getting people to trim recipients is a hopeless battle. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 307 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-dev/attachments/20041110/793754d5/attachment.pgp From foom at fuhm.net Wed Nov 10 18:39:45 2004 From: foom at fuhm.net (James Y Knight) Date: Wed Nov 10 18:39:53 2004 Subject: [Python-Dev] Synchronous and Asynchronous servers in the standard library In-Reply-To: <4191C4F6.2040604@v.loewis.de> References: <20041109071230.20649.1802519922.divmod.quotient.445@ohm> <4191C4F6.2040604@v.loewis.de> Message-ID: <82DA3EDE-333F-11D9-9244-000A95A50FB2@fuhm.net> On Nov 10, 2004, at 2:36 AM, Martin v. L?wis wrote: > I fail to see the problem. If the file object is replaced/rewrapped > into a SSL object, the base class can continue to use > read/write/send/recv - if only the SSL object supported the same > operations (which it really should if switching from plain sockets > to SSL sockets is a common task) (and you seem to be suggesting that > as a good strategy). It can't work completely transparently; SSL has some more complicated requirements. I don't think it's really useful to go into SSL's inner workings here, but, a major issue is that reading from a socket can sometimes require writing to it and vice versa. > My point here is that you just cannot use the very same class for > SMTP, and SMTP+STARTTLS - you indicated that a subclass must be > created. This was my original point: you cannot instantiate the > baseclass for use with TLS, as the baseclass does not implement > the cmd_STARTTLS method. False. STARTTLS is part of the SMTP protocol and is always implemented, as shown in JP's message. Therefore, there is no subclass, STARTTLS is part of the main SMTP class. But, the real point is that you can use the same class for SMTP and SMTP-over-ssl, which is fortunate, because you need to switch to the over-ssl implementation half way through the connection. This discussion really seems useless to me. Asyncore certainly has some serious deficiencies, and arguing each and every one of them is okay because alternatively "nobody needs that feature" or "If X worked in some way (which it doesn't), then everything would be fine" just seems silly. I've only started developing with Twisted fairly recently, so I don't know the entire history of its development, but I'm pretty sure it wasn't started just for fun... James From jcarlson at uci.edu Wed Nov 10 19:38:06 2004 From: jcarlson at uci.edu (Josiah Carlson) Date: Wed Nov 10 19:46:46 2004 Subject: [Python-Dev] Synchronous and Asynchronous servers in the standard library In-Reply-To: <82DA3EDE-333F-11D9-9244-000A95A50FB2@fuhm.net> References: <4191C4F6.2040604@v.loewis.de> <82DA3EDE-333F-11D9-9244-000A95A50FB2@fuhm.net> Message-ID: <20041110100357.F8BE.JCARLSON@uci.edu> James Y Knight <foom@fuhm.net> wrote: > > On Nov 10, 2004, at 2:36 AM, Martin v. L?wis wrote: > > My point here is that you just cannot use the very same class for > > SMTP, and SMTP+STARTTLS - you indicated that a subclass must be > > created. This was my original point: you cannot instantiate the > > baseclass for use with TLS, as the baseclass does not implement > > the cmd_STARTTLS method. > > False. STARTTLS is part of the SMTP protocol and is always implemented, > as shown in JP's message. Therefore, there is no subclass, STARTTLS is > part of the main SMTP class. False. ESMTP has /optional/ support for STARTTLS, as defined in RFC 3207. Neither SMTP nor ESMTP compliant mail servers need to implement STARTTLS, regardless of their compliance to SMTP or ESMTP. > But, the real point is that you can use the same class for SMTP and > SMTP-over-ssl, which is fortunate, because you need to switch to the > over-ssl implementation half way through the connection. You statement is a no-op. One could implement the greater part of most any application in a single class, the question is about how much can be shared. At the minimal level, self.send() and self.recv() need to have different implementations for the different with/without SSL variants (assuming one would merely wrap the bare socket). No surprise there. At a higher level, because SSL-ifying a socket is a blocking operation (requires at least one round-trip if I remember correctly), discussion about async SMTP+TLS is pretty moot at this point (unless one uses/abuses tlslite). > This discussion really seems useless to me. Asyncore certainly has some > serious deficiencies, and arguing each and every one of them is okay > because alternatively "nobody needs that feature" or "If X worked in > some way (which it doesn't), then everything would be fine" just seems > silly. I've only started developing with Twisted fairly recently, so I > don't know the entire history of its development, but I'm pretty sure > it wasn't started just for fun... Asyncore was written to scratch Sam Rushing's itch. I'm not up on Twisted's history. - Josiah From bob at redivi.com Wed Nov 10 20:16:11 2004 From: bob at redivi.com (Bob Ippolito) Date: Wed Nov 10 20:16:21 2004 Subject: [Python-Dev] Synchronous and Asynchronous servers in the standard library In-Reply-To: <20041110100357.F8BE.JCARLSON@uci.edu> References: <4191C4F6.2040604@v.loewis.de> <82DA3EDE-333F-11D9-9244-000A95A50FB2@fuhm.net> <20041110100357.F8BE.JCARLSON@uci.edu> Message-ID: <FB63B6FA-334C-11D9-B23B-000A95BA5446@redivi.com> On Nov 10, 2004, at 13:38, Josiah Carlson wrote: > James Y Knight <foom@fuhm.net> wrote: >> >> But, the real point is that you can use the same class for SMTP and >> SMTP-over-ssl, which is fortunate, because you need to switch to the >> over-ssl implementation half way through the connection. > > You statement is a no-op. One could implement the greater part of most > any application in a single class, the question is about how much can > be > shared. > > At the minimal level, self.send() and self.recv() need to have > different > implementations for the different with/without SSL variants (assuming > one would merely wrap the bare socket). No surprise there. At a > higher > level, because SSL-ifying a socket is a blocking operation (requires at > least one round-trip if I remember correctly), discussion about async > SMTP+TLS is pretty moot at this point (unless one uses/abuses tlslite). Depends on what you mean by blocking. As far as Twisted implementations are concerned, nothing is supposed to block Python code from executing unless it has nothing to do and it's sitting on a select(...) or equivalent (depends on which reactor you are using). Such round trip operations are typically done with a Deferred: a placeholder for a success value or failure exception (wrapped in a Failure instance) that will be available sometime in the future. When you call some function that needs to do a "blocking" operation, it will return a Deferred object. Once you have that Deferred object you can attach a chain of success / failure callbacks to it. When the producer of the Deferred object is ready to do so, it will start the callback chain by passing the Deferred object a the success or failure value. That value (well, the return value of the previous callback in the chain) will be passed along the chain until the chain ends. The added bonus to using Deferreds is that they're not just for sockets. Deferreds can be used to get the return value of a *blocking* function by using the "callInThread" mechanism, which dispatches this function to a thread from the reactor's thread pool and returns a Deferred object that will be passed the return value of that function. You can also use them to get the result of some long-running computation (that has been broken into small chunks, executing once per runloop iteration by way of a timer). >> This discussion really seems useless to me. Asyncore certainly has >> some >> serious deficiencies, and arguing each and every one of them is okay >> because alternatively "nobody needs that feature" or "If X worked in >> some way (which it doesn't), then everything would be fine" just seems >> silly. I've only started developing with Twisted fairly recently, so I >> don't know the entire history of its development, but I'm pretty sure >> it wasn't started just for fun... > > Asyncore was written to scratch Sam Rushing's itch. I'm not up on > Twisted's history. I can't speak for its original raison d'etre, but I moved from asyncore to Twisted because I ran into limitations in asyncore that Twisted had already taken care of. As an added bonus, there was a larger more responsive community behind it, and a lot of existing protocol implementations (most if not all of which are more complete than any standard library counterpart, if one exists) that came standard with Twisted I could piggy-back off of. Originally I had stayed away from Twisted because it was big and complicated in comparison to asyncore, but at some point a light went off in my head and I realized that it would make my applications SIGNIFICANTLY shorter, had methods for doing the things I needed to do that asyncore didn't do well, and solved some problems I didn't even know I would have. -bob From foom at fuhm.net Wed Nov 10 20:18:38 2004 From: foom at fuhm.net (James Y Knight) Date: Wed Nov 10 20:18:57 2004 Subject: [Python-Dev] Synchronous and Asynchronous servers in the standard library In-Reply-To: <20041110100357.F8BE.JCARLSON@uci.edu> References: <4191C4F6.2040604@v.loewis.de> <82DA3EDE-333F-11D9-9244-000A95A50FB2@fuhm.net> <20041110100357.F8BE.JCARLSON@uci.edu> Message-ID: <534381F4-334D-11D9-9244-000A95A50FB2@fuhm.net> On Nov 10, 2004, at 1:38 PM, Josiah Carlson wrote: > False. ESMTP has /optional/ support for STARTTLS, as defined in RFC > 3207. Neither SMTP nor ESMTP compliant mail servers need to implement > STARTTLS, regardless of their compliance to SMTP or ESMTP. That is irrelevant. My point was, there is no need to make a subclass for STARTTLS. There is no architectural reason that it couldn't be in the same place as the other supported ESMTP commands. However, the original comment, and my response, and your response are all besides the main point, which is about SSL vs non-SSL sockets, not about whether you should have a separate class for SMTP and SMTP+some optional extensions. >> But, the real point is that you can use the same class for SMTP and >> SMTP-over-ssl, which is fortunate, because you need to switch to the >> over-ssl implementation half way through the connection. > > You statement is a no-op. One could implement the greater part of most > any application in a single class, the question is about how much can > be > shared. > > At the minimal level, self.send() and self.recv() need to have > different > implementations for the different with/without SSL variants (assuming > one would merely wrap the bare socket). No surprise there. The point is that the above assumption is a poor one. The protocol should not be wrapping the bare socket. With an architecture like Twisted, your protocol class doesn't have implementations of send and recv. Those belong to the transport, which you call. So, you do not have to derive from a different base class for SMTP-over-SSL and SMTP-over-TCP, and then do yucky things like switching your class halfway through the connection. > At a higher > level, because SSL-ifying a socket is a blocking operation (requires at > least one round-trip if I remember correctly), discussion about async > SMTP+TLS is pretty moot at this point (unless one uses/abuses tlslite). That is completely incorrect. OpenSSL works perfectly well in async mode. Twisted implements async SSL __right now__. Again, I do not think this is the appropriate place to be giving SSL lessons, but, yes, at least one roundtrip is required to setup an SSL session. But, this does __not__ mean your code has to block waiting for the roundtrip to complete. OpenSSL returns with an error code if it needs more read/write data to complete an operation. You get to take that information and feed it to select to wait for the condition to be fulfilled, and call OpenSSL again. This works. James From skip at pobox.com Wed Nov 10 20:57:41 2004 From: skip at pobox.com (Skip Montanaro) Date: Wed Nov 10 20:57:49 2004 Subject: [Python-Dev] Re: [development doc updates] In-Reply-To: <20041110154700.5763718E89A@grendel.fdrake.net> References: <20041110154700.5763718E89A@grendel.fdrake.net> Message-ID: <16786.29365.808646.948039@montanaro.dyndns.org> Fred> Lots of changes to how tables are actually formatted, with much of Fred> the styling moved from the HTML to the CSS stylesheet. Fred> Tables also look nicer in supporting browsers. Any idea what browsers those would be? Fred> Please review the tables and report any strange presentations; be Fred> sure to include the specific browser (including platform and Fred> version!) in your report. Using Firefox 1.0PR on a Solaris system I checked out http://www.python.org/dev/doc/devel/lib/boolean.html There are no left, right or top borders. A PNG is at http://www.musi-cal.com/~skip/table.png Skip From fdrake at acm.org Wed Nov 10 21:33:23 2004 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Wed Nov 10 21:33:33 2004 Subject: [Python-Dev] Re: [development doc updates] In-Reply-To: <16786.29365.808646.948039@montanaro.dyndns.org> References: <20041110154700.5763718E89A@grendel.fdrake.net> <16786.29365.808646.948039@montanaro.dyndns.org> Message-ID: <200411101533.23469.fdrake@acm.org> On Wednesday 10 November 2004 02:57 pm, Skip Montanaro wrote: > Fred> Lots of changes to how tables are actually formatted, with much > of Fred> the styling moved from the HTML to the CSS stylesheet. > > Fred> Tables also look nicer in supporting browsers. > > Any idea what browsers those would be? All graphical browsers I tested with until I booted a Windows machine. ;-) Specifically: Mozilla 1.8a4 on Linux, Opera 7.21 on Linux, Galeon 1.2.13 (yet another Gecko wrapper) on Linux. Things looked fine on links, lynx, and w3m as well, though I don't expect they looked better than before. ;-) > Fred> Please review the tables and report any strange presentations; > be Fred> sure to include the specific browser (including platform and > Fred> version!) in your report. > > Using Firefox 1.0PR on a Solaris system I checked out > > http://www.python.org/dev/doc/devel/lib/boolean.html > > There are no left, right or top borders. A PNG is at That's right. I liked that better myself; fewer extra lines. The lines below the header and at the bottom of the table are slightly heavier than the other lines as well. I'm surprised that the first and last columns aren't centered, and that the heading for the second column is. I'll need to review more of the actual tables in the documents, though the tables I used are quite similar to this one. -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> From bac at OCF.Berkeley.EDU Thu Nov 11 00:47:42 2004 From: bac at OCF.Berkeley.EDU (Brett C.) Date: Thu Nov 11 00:47:57 2004 Subject: [Python-Dev] python-dev Summary for 2004-10-01 through 2004-10-15 [draft] Message-ID: <4192A89E.1090305@ocf.berkeley.edu> No, that subject does not have a typo; this is for the first half of October. Have two huge projects due at the end of the quarter and they are killing me. Plus I have doctoral apps to prep for and do (and suggestions on good schools in the US or even Canada would be great). Anyway, expect major delays in getting more Summaries out of the door until I am done with this quarter in December. ----------------------------------- ===================== Summary Announcements ===================== Sorry for the very late and short summary but I am in extreme crunch mode right now with school and preparing for applications to doctorate programs (any recommendations for schools with a good CS dept. and language research would be appreciated). Don't expect things to pick up again until the quarter is over (Dec. 9th is my last final). ========= Summaries ========= ---------------------- New module: subprocess ---------------------- Peter ?strand wrote `PEP 324`_ to come up with a platform-independent way to create processes. Originally entitled popen5, it has now been added to the stdlib as subprocess. .. _PEP 324: http://www.python.org/peps/pep-0324.html Contributing threads: - `subprocess - updated popen5 module <>`__ - `python/dist/src/Lib subprocess.py,NONE,1.1 <>`__ - `Wither docs for the subprocess module? <>`__ -------------------- Python 2.4.b1 is out -------------------- `Python 2.4b1`__ has been released. As usual please download it, run the test suite, and test it with your own code. Contributing threads: - `2.4b1 release, this Friday (AU time) -- imminent TRUNK FREEZING <>`__ - `RELEASED Python 2.4, beta 1 <>`__ - `TRUNK UNFROZEN (release is done) <>`__ -------------------------------------- We need a replacement for ConfigParser -------------------------------------- David Goodger brought forward two patches for ConfigParser he wanted to apply. This quickly brought up the point that ConfigParser's API is not exactly optimal. So the decision was made to field new designs. If you happen to have your own code lying around that handles .ini files nicely then get it cleaned up and consider proposing at as a replacement. Some ideas have already been floated on the list but will have to wait for the next summary for mention/discussion. Contributing threads: - `ConfigParser patches <>`__ =============== Skipped Threads =============== - Guidelines for Logging Usage - Status of undocumented distutils features in 2.4? - Toward Python's future article - test_pep263.py is "-kb" in CVS - python/dist/src/Doc/whatsnew whatsnew24.tex, 1.108, 1.109 - Python on Windows 64bits? - About the Wise installer on Windows - Patch wanted - python/dist/src/Objects unicodeobject.c, 2.228, 2.229 - Minor cookielib changes for 2.4b1 - proposed struct module format code addition - Cyclic GC issues Just don't have the time to summarize this, but a in-depth discussion of garbage collection involving extension code and using non-Python structs. - python-dev Summary for 2004-09-16 through 2004-09-30 [draft] Lead to a discussion of executing modules in packages using the new '-m' command-line flag. From db3l at fitlinxx.com Thu Nov 11 02:28:46 2004 From: db3l at fitlinxx.com (David Bolen) Date: Thu Nov 11 02:28:54 2004 Subject: [Python-Dev] Re: Synchronous and Asynchronous servers in the standard library References: <20041107133719.F86E.JCARLSON@uci.edu> <Pine.LNX.4.10.10411071531210.25017-100000@sumeru.stanford.EDU> <20041108001901.GB477@frobozz> <418F0921.5040905@v.loewis.de> <6278459D-318D-11D9-ADE7-000A95EFAE9E@yahoo.com> <41905A52.2080507@v.loewis.de> <u4qjya96d.fsf@fitlinxx.com> <4191C538.4050209@v.loewis.de> Message-ID: <uvfcdnpgx.fsf@fitlinxx.com> "Martin v. L?wis" <martin@v.loewis.de> writes: > David Bolen wrote: > > On Windows, I miss the ability to mix in serial port I/O the most, > > since many of my servers are translators between the network and > > serial devices. > > Do you know whether IOCP could be used with the serial ports? I normally use overlapped I/O with my own WaitForMultipleObjects loop, but IOCP should work just as well, since access to a serial port is just another system handle (the port is opened with CreateFile just like normal files) under Windows and you just use Read/WriteFile to process the data stream. IMHO, Windows is arguably more consistent across multiple types of objects than Unix's "everything is a file" approach. With something like Windows WaitForMultipleObjects you can include handles to system objects for files, semaphores, events, processes and threads, timers, etc... many of which are concepts that don't fall neatly into a select() call under Unix. The rub of course is that sockets are the odd man out, and really feel bolted on afterwards in Windows (you can read/write to them as file handles, but not select them directly with the Wait* operations). If you're just staying with Windows, you can map sockets to a compatible object to use with other system objects, but going in through select() locks you in to sockets only. For my part, in terms of integrating non-socket sources, I'd probably be just as happy if Python permitted it's own Queue.Queue to be selectable, since then I could just have a separate thread with a native Win32 wait object loop, pumping information into the queue. But the basic nature of the fundamantal threading constructs Queue is built on would probably make that difficult, if possible at all. -- David From exarkun at divmod.com Thu Nov 11 03:17:40 2004 From: exarkun at divmod.com (Jp Calderone) Date: Thu Nov 11 03:17:59 2004 Subject: [Python-Dev] Re: Synchronous and Asynchronous servers in the standard library In-Reply-To: <uvfcdnpgx.fsf@fitlinxx.com> Message-ID: <20041111021740.20649.160888204.divmod.quotient.2232@ohm> On 10 Nov 2004 20:28:46 -0500, David Bolen <db3l@fitlinxx.com> wrote: >"Martin v. L�wis" <martin@v.loewis.de> writes: > > [snip] > > For my part, in terms of integrating non-socket sources, I'd probably > be just as happy if Python permitted it's own Queue.Queue to be > selectable, since then I could just have a separate thread with a > native Win32 wait object loop, pumping information into the queue. > But the basic nature of the fundamantal threading constructs Queue is > built on would probably make that difficult, if possible at all. This isn't very difficult. See _UnixWaker/_Win32Waker at in http://cvs.twistedmatrix.com./cvs/trunk/twisted/internet/default.py?view=markup&rev=12283&root=Twisted Building a "selectable Queue" on top of the "wakeUp" primitive is a handful of lines at most. Of course, with the deferToThread() function already built on top of it, the selectable Queue concept doesn't seem quite as attractive anymore. Jp From kbk at shore.net Thu Nov 11 05:01:21 2004 From: kbk at shore.net (Kurt B. Kaiser) Date: Thu Nov 11 05:01:27 2004 Subject: [Python-Dev] Weekly Python Patch/Bug Summary Message-ID: <200411110401.iAB41LE3007062@h006008a7bda6.ne.client2.attbi.com> Patch / Bug Summary ___________________ Patches : 244 open ( +3) / 2698 closed (+15) / 2942 total (+18) Bugs : 771 open (-13) / 4595 closed (+28) / 5366 total (+15) RFE : 157 open ( -1) / 134 closed ( +3) / 291 total ( +2) New / Reopened Patches ______________________ add missing __all__ to pickletools.py (2004-11-07) CLOSED http://python.org/sf/1061679 opened by George Yoshida fix for bug 1055168 (pdb not entering the right frame) (2004-11-06) CLOSED http://python.org/sf/1061767 opened by Ilya Sandler use a new -m option in pdb and profile invocations (2004-11-06) CLOSED http://python.org/sf/1061780 opened by Ilya Sandler Source code encoding in IDLE console (2004-11-07) http://python.org/sf/1061803 opened by Hye-Shik Chang [ 1061857 ] Error and omission in logging docs (2004-11-07) CLOSED http://python.org/sf/1061882 opened by Jeroen Vloothuis fix for bug 878275 class MyTest(unittest.TestSuite) (2004-11-07) CLOSED http://python.org/sf/1061904 opened by Irmen de Jong Relative to bug #1061857 -- logging docs (2004-11-07) CLOSED http://python.org/sf/1061924 opened by Rodrigo Dias Arruda Senra [ 971872 ] An apparent bug in help (2004-11-07) CLOSED http://python.org/sf/1061931 opened by Jeroen Vloothuis fix for 988120 httplib.py not updating length correctly (2004-11-07) CLOSED http://python.org/sf/1061941 opened by Irmen de Jong fix for 764437 AF_UNIX socket special linux socket names (2004-11-07) http://python.org/sf/1062014 opened by Irmen de Jong Fix for bug issue #1038693 -- bad link in Tkinter docs (2004-11-07) CLOSED http://python.org/sf/1062018 opened by Rodrigo Dias Arruda Senra Half-fix for 1027771 - DOM insertBefore (2004-11-07) CLOSED http://python.org/sf/1062033 opened by Rodrigo Dias Arruda Senra fix for 1016880 urllib.urlretrieve silently truncates dwnld (2004-11-07) http://python.org/sf/1062060 opened by Irmen de Jong deque pickling problems (2004-11-08) CLOSED http://python.org/sf/1062279 opened by Dima Dorfman set pickling problems (2004-11-08) CLOSED http://python.org/sf/1062353 opened by Dima Dorfman Pickle breakage with reduction of recursive structures (2004-11-08) http://python.org/sf/1062277 opened by Dima Dorfman distutils and python 1.5.2 (2004-11-09) CLOSED http://python.org/sf/1063059 opened by Davide Alberani Tkinter clipboard_get method (new in Tk 8.4) (2004-11-10) http://python.org/sf/1063914 opened by Martin Franklin Patches Closed ______________ add missing __all__ to pickletools.py (2004-11-06) http://python.org/sf/1061679 closed by tim_one fix for bug 1055168 (pdb not entering the right frame) (2004-11-07) http://python.org/sf/1061767 closed by jlgijsbers use a new -m option in pdb and profile invocations (2004-11-07) http://python.org/sf/1061780 closed by rhettinger [ 1061857 ] Error and omission in logging docs (2004-11-07) http://python.org/sf/1061882 closed by jlgijsbers fix for bug 878275 class MyTest(unittest.TestSuite) (2004-11-07) http://python.org/sf/1061904 closed by jlgijsbers Check for NULL returns in compile.c:com_import_stmt (2004-09-10) http://python.org/sf/1025636 closed by jhylton Relative to bug #1061857 -- logging docs (2004-11-07) http://python.org/sf/1061924 closed by jlgijsbers [ 971872 ] An apparent bug in help (2004-11-07) http://python.org/sf/1061931 closed by jlgijsbers fix for 988120 httplib.py not updating length correctly (2004-11-07) http://python.org/sf/1061941 closed by jhylton Update HTTPRespnse.length (2004-07-10) http://python.org/sf/988642 closed by jhylton Fix for bug issue #1038693 -- bad link in Tkinter docs (2004-11-07) http://python.org/sf/1062018 closed by jlgijsbers Half-fix for 1027771 - DOM insertBefore (2004-11-07) http://python.org/sf/1062033 closed by jlgijsbers deque pickling problems (2004-11-08) http://python.org/sf/1062279 closed by rhettinger set pickling problems (2004-11-08) http://python.org/sf/1062353 closed by rhettinger distutils and python 1.5.2 (2004-11-09) http://python.org/sf/1063059 closed by loewis New / Reopened Bugs ___________________ test_socket fails (2004-11-06) http://python.org/sf/1061429 opened by TPJ spelling error in win installer (2004-11-06) CLOSED http://python.org/sf/1061457 opened by Simon Dahlbacka Manual typesets bit-shift operators as guillemet (2004-11-06) http://python.org/sf/1061770 opened by Brion Vibber Error and omission in logging docs (2004-11-07) CLOSED http://python.org/sf/1061857 opened by Kent Johnson "k" specifier in PyArg_ParseTuple incomplete documentated (2004-11-07) http://python.org/sf/1061920 opened by Gustavo J. A. M. Carneiro threads: segfault or Py_FatalError at exit (2004-11-07) CLOSED http://python.org/sf/1061968 opened by Armin Rigo test_subprocess fails on winXP (2004-11-07) CLOSED http://python.org/sf/1061991 opened by Miki Tebeka locale.getdefaultlocale() returns wrong encoding (2004-11-08) CLOSED http://python.org/sf/1062261 opened by Andreas Jung pwent objects from the pwd module ar e not pickle-able (2004-11-08) http://python.org/sf/1062708 opened by Keith Dart not enough information in SGMLParserError? (2004-11-09) http://python.org/sf/1063229 opened by Alan Ezust test_subprocess 2.4b2 fails on tru64 (2004-11-10) http://python.org/sf/1063571 opened by roadkill Modules/zipimport.c does not compile on solaris (2004-11-08) CLOSED http://python.org/sf/1062495 opened by Niki W. Waibel optparse docs mildly mangled (2004-11-10) http://python.org/sf/1063757 opened by Michael Hudson IDLE aborts on any error when started with "-n" (2004-11-10) http://python.org/sf/1063840 opened by Miki Tebeka asyncore should handle ECONNRESET in send (2004-11-10) http://python.org/sf/1063924 opened by Florent Guillaume Random core dumps (2004-11-10) http://python.org/sf/1063937 opened by munder12 Bugs Closed ___________ spelling error in win installer (2004-11-06) http://python.org/sf/1061457 closed by rhettinger Why does Python link to Foundation? (2004-09-24) http://python.org/sf/1034277 closed by bcannon time module returns wrong timezone under windows (2004-10-24) http://python.org/sf/1053539 closed by bcannon incorrect "'yield' outside function" error (2004-11-04) http://python.org/sf/1060135 closed by rhettinger compiler.transformer, "from module import *" (2004-10-31) http://python.org/sf/1057835 closed by mwh pdb.set_trace should go up one more frame (2004-10-27) http://python.org/sf/1055168 closed by jlgijsbers Buffer overflow in socketmodule.c (2004-11-04) http://python.org/sf/1060396 closed by jhylton test_unicode_file fails on win2k (2004-07-12) http://python.org/sf/989338 closed by loewis 2.4b1 Win32 installer eliminates 2.3 assoc (2004-10-20) http://python.org/sf/1050486 closed by loewis Error and omission in logging docs (2004-11-07) http://python.org/sf/1061857 closed by jlgijsbers Implicit close() should check for errors (2004-05-24) http://python.org/sf/959379 closed by astrand subprocess on Windows: extra windows (2004-10-29) http://python.org/sf/1057061 closed by astrand An apparent bug in help (2004-06-12) http://python.org/sf/971872 closed by jlgijsbers httplib.HTTPResponse.read() not updating the length correctl (2004-07-09) http://python.org/sf/988120 closed by jhylton subprocess on Windows 2: unexpected failure (2004-10-29) http://python.org/sf/1057052 closed by astrand threads: segfault or Py_FatalError at exit (2004-11-07) http://python.org/sf/1061968 closed by arigo subprocess fails for args="...", executable="..." (2004-10-29) http://python.org/sf/1056441 closed by astrand test_subprocess fails on winXP (2004-11-07) http://python.org/sf/1061991 closed by tebeka bad link in Tkinter docs (2004-10-01) http://python.org/sf/1038693 closed by jlgijsbers file.next() info hidden (2004-09-29) http://python.org/sf/1036626 closed by jlgijsbers In DOM Node Objects, add more explanations for insertBefore (2004-09-14) http://python.org/sf/1027771 closed by jlgijsbers httplib throws a TypeError when the target host disconnects (2002-08-06) http://python.org/sf/591349 closed by jlgijsbers locale.getdefaultlocale() returns wrong encoding (2004-11-08) http://python.org/sf/1062261 closed by jlgijsbers senddigest error (2004-09-01) http://python.org/sf/1020605 closed by facundobatista Memory leakage in SAX with exception (2002-07-31) http://python.org/sf/589149 closed by facundobatista Include/pyport.h: Bad LONG_BIT assumption on non-glibc sys (2004-09-07) http://python.org/sf/1023838 closed by cdgregorr test_xrange fails on osf1 v5.1b (2004-09-06) http://python.org/sf/1022813 closed by dharma_roadkill Modules/zipimport.c does not compile on solaris (2004-11-08) http://python.org/sf/1062495 closed by rhettinger New / Reopened RFE __________________ Remove troublesome assert in trace.py (2004-11-07) CLOSED http://python.org/sf/1062190 opened by Jp Calderone RFE Closed __________ lock-free data structures? (2004-11-03) http://python.org/sf/1059545 closed by tim_one optparse error method doesn't print option info (2004-07-29) http://python.org/sf/1000439 closed by nestler Remove troublesome assert in trace.py (2004-11-07) http://python.org/sf/1062190 closed by rhettinger From leogah at spamcop.net Wed Nov 10 22:37:53 2004 From: leogah at spamcop.net (Richard Brodie) Date: Thu Nov 11 05:23:53 2004 Subject: [Python-Dev] Re: [development doc updates] Message-ID: <000301c4c76d$89518400$af0189c3@oemcomputer> > Fred L. Drake, Jr. wrote > I'm surprised that the first and last columns aren't centered, and that the > heading for the second column is. It seems to me that: .realtable th {text-align: inherit} is more specific than .center {text-align: center} , so the second rule needs strengthening. Richard Brodie. From fdrake at acm.org Thu Nov 11 05:42:06 2004 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Thu Nov 11 05:42:30 2004 Subject: [Python-Dev] Re: [development doc updates] In-Reply-To: <000301c4c76d$89518400$af0189c3@oemcomputer> References: <000301c4c76d$89518400$af0189c3@oemcomputer> Message-ID: <200411102342.06528.fdrake@acm.org> On Wednesday 10 November 2004 04:37 pm, Richard Brodie wrote: > It seems to me that: > > .realtable th {text-align: inherit} is more specific than > .center {text-align: center} , > > so the second rule needs strengthening. Bingo! I need the second rule in its simple form elsewhere, but I can actually remove the first completely since I set the alignment-controlling class for all cells in a .realtable. The first of those rules was added while I was trying to make <col> useful for column layout, but browsers haven't implemented that sufficiently to do so yet. I've tested the fix with my browsers and committed to CVS; an updated version of the documentation should be online later this evening. -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> From whisper at oz.net Thu Nov 11 21:58:48 2004 From: whisper at oz.net (whisper@oz.net) Date: Thu Nov 11 21:58:51 2004 Subject: [Python-Dev] @boo @hiss def NewDeclSyntaxSux(): Message-ID: <4193D288.7020405@oz.net> Hmm... wander off for a bit and come back to find that Python has taken a big step towards being an Ugly Language[tm]. I can hardly wait for the advent of $ and # as some sort of syntactial marker. Being in beta, it's obviously too late to do anything about it, but I think this is going to turn out to be a REALLY BAD MOVE. David LeBlanc Seattle, WA USA From aahz at pythoncraft.com Thu Nov 11 22:38:45 2004 From: aahz at pythoncraft.com (Aahz) Date: Thu Nov 11 22:38:48 2004 Subject: [Python-Dev] @boo @hiss def NewDeclSyntaxSux(): In-Reply-To: <4193D288.7020405@oz.net> References: <4193D288.7020405@oz.net> Message-ID: <20041111213845.GA13333@panix.com> On Thu, Nov 11, 2004, whisper@oz.net wrote: > > Hmm... wander off for a bit and come back to find that Python has taken > a big step towards being an Ugly Language[tm]. > > I can hardly wait for the advent of $ and # as some sort of syntactial > marker. > > Being in beta, it's obviously too late to do anything about it, but I > think this is going to turn out to be a REALLY BAD MOVE. While I agree with you, please remember that python-dev is a mailing list where language should stay at the professional level. If you want to emote, please use comp.lang.python instead. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ WiFi is the SCSI of the 21st Century -- there are fundamental technical reasons for sacrificing a goat. (with no apologies to John Woods) From tim.peters at gmail.com Fri Nov 12 05:29:03 2004 From: tim.peters at gmail.com (Tim Peters) Date: Fri Nov 12 05:29:06 2004 Subject: [Python-Dev] Code coverage tool updated In-Reply-To: <4191FDB2.2000002@livinglogic.de> References: <41864572.4000806@livinglogic.de> <16774.45652.835103.327656@montanaro.dyndns.org> <4187765D.3000705@livinglogic.de> <4187CA1C.9050508@livinglogic.de> <1f7befae04110210537c35e514@mail.gmail.com> <418A269B.6010207@livinglogic.de> <1f7befae0411051406711618cf@mail.gmail.com> <4191FDB2.2000002@livinglogic.de> Message-ID: <1f7befae04111120293d9c489b@mail.gmail.com> [Walter D?rwald] > Using regrtest.py's tracing option instead of trace.py gives better > results (see http://styx.livinglogic.de/~walter/brokentrace2.txt): Cool! > 656 covered modules (including a strange /tmp/tmp6ccfd4/t5/string.py) test_pkg.py creates the weird string.py. Don't know why. > There seems to be something wrong with the way trace.py starts > the script. > > BTW, I'd like to have an option the specify the coverdir in regrtest.py, > as now finding the coverage file when you have the name of the Python > file is more complicated. But I guess this has to wait until 2.4 is > out the door. I don't know anything about this, but there's no reason to avoid changing regrtest.py just because 2.4 is in beta. The arguments to regrtest.py aren't documented, and it's an internal module to help us test Python anyway. So if you can do better 2.4 testing by changing it now, change it now. From tim.peters at gmail.com Fri Nov 12 05:51:15 2004 From: tim.peters at gmail.com (Tim Peters) Date: Fri Nov 12 05:51:17 2004 Subject: [Python-Dev] Deprecate PyRange_New() In-Reply-To: <000001c4c6e6$cda46fa0$6628a044@oemcomputer> References: <000001c4c6e6$cda46fa0$6628a044@oemcomputer> Message-ID: <1f7befae04111120514932c693@mail.gmail.com> [Raymond Hettinger] > A while back, Tim added a comment that PyRange_New() should be > deprecated in-part because it is filled with problems and is not used > anywhere in the core. And because it's undocumented. The only place it's even mentioned is in the NEWS for 2.3a1: - PyRange_New() now raises ValueError if the fourth argument is not 1. This is part of the removal of deprecated features of the xrange object. > Doesn't anyone mind if I mark it in the C-API docs as deprecated so it > can be phased out later? +1 I don't object to *documenting* a PyRange_New() API function, BTW, but the current implementation is too broken to fix with less than heroic effort. If PyRange_New() needs to exist in the C API, then it should have the same (lo, hi, step) signature as the builtin range(), for which we have correct but non-obvious error-checking C code. The funky (start, len, step, reps) signature of the current PyRange_New() is much more difficult to check for errors (e.g., detecting C long multiplication overflow in portable C is a sub-problem), and the current signature is too surprising regardless (because it's so different from the builtin range()). From FBatista at uniFON.com.ar Fri Nov 12 13:48:23 2004 From: FBatista at uniFON.com.ar (Batista, Facundo) Date: Fri Nov 12 13:50:19 2004 Subject: [Python-Dev] Floating point -> string conversions Message-ID: <A128D751272CD411BC9200508BC2194D053C7BC0@escpl.tcp.com.ar> There's a thread in the standard list (with this very Subject) that talks about implicit coercing with "%d" in strings replacement. jfouhy posted this example: >>> pow(2, 31) 2147483648L >>> '%d' % 2147483647.0 # python will convert to int '2147483647' >>> '%d' % 2147483648.0 # too big for an int, so error. Traceback (most recent call last): File "<stdin>", line 1, in ? TypeError: int argument required >>> '%d' % long(2147483648.0) # but yet, no trouble accepting a long. '2147483648' >>> '%d' % int(2147483648.0) # and int() converts to long anyway '2147483648' I think that here's a bug, but don't know where: Should %d accept only integer values? So this is a bug: >>> '%d' % 2147483647.0 '2147483647' Should %d accept also floats, and make the conversion? So there're two bugs: - In the doc (it's not explicited that it should accept a other data types) - In this behaviour difference: >>> '%d' % 2147483647.0 '2147483647' >>> '%d' % 2147483648.0 Traceback (most recent call last): ... TypeError: int argument required I really don't know about opening a bug, because I don't know which to open, :p . Facundo -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20041112/929d3980/attachment.html From arigo at tunes.org Fri Nov 12 14:12:36 2004 From: arigo at tunes.org (Armin Rigo) Date: Fri Nov 12 14:19:37 2004 Subject: [Python-Dev] struct module: add OverflowErrors now? Message-ID: <20041112131236.GA29540@vicky.ecs.soton.ac.uk> Hi, The struct module is too lazy in checking for overflows (it only checks some cases on some architectures). See patch #1038854. The question is: can we make it strict this late in the 2.4 release process? This *will* break some user code; the proof is that it breaks some standard library code, e.g. gzip.py, which uses a signed typecode when it really means an unsigned one. Fixing it in 2.4 would be nice, but on the other hand we could postpone it for the 2.5 trunk; people trying 2.5 early will see the problem quickly. We could add warnings for 2.4 but that looks overkill. Or we could just go ahead and break existing buggy code with 2.4b3. Opinions? Armin From pinard at iro.umontreal.ca Fri Nov 12 15:51:51 2004 From: pinard at iro.umontreal.ca (=?iso-8859-1?Q?Fran=E7ois?= Pinard) Date: Fri Nov 12 15:51:53 2004 Subject: [Python-Dev] struct module: add OverflowErrors now? In-Reply-To: <20041112131236.GA29540@vicky.ecs.soton.ac.uk> References: <20041112131236.GA29540@vicky.ecs.soton.ac.uk> Message-ID: <20041112145151.GA16397@alcyon.progiciels-bpi.ca> > The struct module is too lazy in checking for overflows [...]. Agreed. Python is usually on the side of not letting errors go through silently, and as we learn to comfortably rely on this behaviour, I found the weakness of `struct' in this area a bit, huh, surprising. :-) -- Fran?ois Pinard http://pinard.progiciels-bpi.ca From aahz at pythoncraft.com Fri Nov 12 16:41:53 2004 From: aahz at pythoncraft.com (Aahz) Date: Fri Nov 12 16:42:00 2004 Subject: [Python-Dev] struct module: add OverflowErrors now? In-Reply-To: <20041112131236.GA29540@vicky.ecs.soton.ac.uk> References: <20041112131236.GA29540@vicky.ecs.soton.ac.uk> Message-ID: <20041112154153.GB5470@panix.com> On Fri, Nov 12, 2004, Armin Rigo wrote: > > The struct module is too lazy in checking for overflows (it only > checks some cases on some architectures). See patch #1038854. The > question is: can we make it strict this late in the 2.4 release > process? This *will* break some user code; the proof is that it > breaks some standard library code, e.g. gzip.py, which uses a signed > typecode when it really means an unsigned one. +1 on fixing it now, assuming you can find an appropriate reviewer. Bugfixing is allowed during beta, and relying on this behavior is a bug. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ WiFi is the SCSI of the 21st Century -- there are fundamental technical reasons for sacrificing a goat. (with no apologies to John Woods) From Scott.Daniels at Acm.Org Fri Nov 12 21:33:29 2004 From: Scott.Daniels at Acm.Org (Scott David Daniels) Date: Fri Nov 12 21:32:12 2004 Subject: [Python-Dev] Re: Code coverage tool updated In-Reply-To: <4191FDB2.2000002@livinglogic.de> References: <41864572.4000806@livinglogic.de> <16774.45652.835103.327656@montanaro.dyndns.org> <4187765D.3000705@livinglogic.de> <4187CA1C.9050508@livinglogic.de> <1f7befae04110210537c35e514@mail.gmail.com> <418A269B.6010207@livinglogic.de> <1f7befae0411051406711618cf@mail.gmail.com> <4191FDB2.2000002@livinglogic.de> Message-ID: <cn36k7$jjg$1@sea.gmane.org> Walter D?rwald wrote: >... There seems to be something wrong with the way trace.py starts > the script. I'm starting to loook at writing a dinky trace.py note/document as a result of bug day, and I have seen two problems so far. First, the assert you removed, and second in class CoverageResults: class CoverageResults: def __init__(self, counts=None, calledfuncs=None, infile=None, callers=None, outfile=None): ... try: counts, calledfuncs, callers = \ pickle.load(open(self.infile, 'rb')) self.update(self.__class__(counts, calledfuncs, callers)) ... I suspect switching infile and callers in the parameter list will give better results. I don;t know enough about trace yet to build a test for this problem. -- Scott David Daniels Scott.Daniels@Acm.Org From bac at OCF.Berkeley.EDU Fri Nov 12 22:28:13 2004 From: bac at OCF.Berkeley.EDU (Brett C.) Date: Fri Nov 12 22:28:25 2004 Subject: [Python-Dev] Floating point -> string conversions In-Reply-To: <A128D751272CD411BC9200508BC2194D053C7BC0@escpl.tcp.com.ar> References: <A128D751272CD411BC9200508BC2194D053C7BC0@escpl.tcp.com.ar> Message-ID: <41952AED.4080900@ocf.berkeley.edu> Batista, Facundo wrote: > There's a thread in the standard list (with this very Subject) that > talks about implicit coercing with "%d" in strings replacement. > > jfouhy posted this example: > > >>> pow(2, 31) > 2147483648L > >>> '%d' % 2147483647.0 # python will convert to int > '2147483647' > >>> '%d' % 2147483648.0 # too big for an int, so error. > Traceback (most recent call last): > File "<stdin>", line 1, in ? > TypeError: int argument required > >>> '%d' % long(2147483648.0) # but yet, no trouble accepting a > long. > '2147483648' > >>> '%d' % int(2147483648.0) # and int() converts to long anyway > '2147483648' > > > I think that here's a bug, but don't know where: > I say it is in the conversion from float not trying it as a long to make it work. I expect the conversion types in string interpolation to convert the argument as needed to meet the conversion type. Since a float can safely be converted to a long that is what should be done since in the eyes of Python int == long. Just think of how printf() handles it; it just interprets the argument as what the conversion type says it is. I think Python should do the same. -Brett From python at ce.anoka.k12.mn.us Fri Nov 12 23:22:02 2004 From: python at ce.anoka.k12.mn.us (Mike Tuller) Date: Fri Nov 12 23:21:57 2004 Subject: [Python-Dev] parse xml Message-ID: <46D2E28D-34F9-11D9-93D6-0003938E6950@ce.anoka.k12.mn.us> I am just starting with python. I have a problem where I want to take a document that is in xml and take the items and enter them into a mysql database. The database part I know, I just need some hints on where to look to be able to parse something like this. <dict> <key>_name</key> <string>VLC</string> <key>info</key> <string>VLC media player 0.7.2, Copyright (c) 1996-2004 VideoLAN.</string> <key>lastModified</key> <date>2004-05-21T15:04:12Z</date> <key>path</key> <string>/Applications/VLC.app</string> <key>version</key> <string>0.7.2</string> </dict> <dict> <key>_name</key> <string>WeatherPop</string> <key>lastModified</key> <date>2003-06-19T05:35:22Z</date> <key>path</key> <string>/Applications/WeatherPop.app</string> <key>version</key> <string>1.8.1</string> </dict> etc.etc.etc. I want to be able to get the values for name and version for each application. I don't need any other information. Can someone give me some ideas on where to look for more information on doing something like this? Mike From just at letterror.com Fri Nov 12 23:38:35 2004 From: just at letterror.com (Just van Rossum) Date: Fri Nov 12 23:38:32 2004 Subject: [Python-Dev] parse xml In-Reply-To: <46D2E28D-34F9-11D9-93D6-0003938E6950@ce.anoka.k12.mn.us> Message-ID: <r01050400-1026-9819104A34FB11D9848D003065D5E7E4@[10.0.0.23]> Mike Tuller wrote: > I am just starting with python. Oops, wrong list. This list is for development OF Python, not WITH. That said, your data below looks a lot like plist. If it is, have a look at: http://cvs.sourceforge.net/viewcvs.py/*checkout*/python/python/dist/src/ Lib/plat-mac/plistlib.py (unwrap url). You need Python 2.3 to use this module. Just From rwgk at cci.lbl.gov Sat Nov 13 05:20:03 2004 From: rwgk at cci.lbl.gov (Ralf W. Grosse-Kunstleve) Date: Sat Nov 13 05:20:09 2004 Subject: [Python-Dev] proposal+patch: sys.gettickeraccumulation() Message-ID: <200411130420.iAD4K3K2006945@boa.lbl.gov> Proposal: >>> sys.gettickeraccumulation.__doc__ 'getgettickeraccumulation() -> current count of bytecodes processed by the interpreter.' Target environments: Python-based systems with (many) extension modules. Motivation: A fast, easy, and non-intrusive method for estimating how much time is spent in Python code and how much time in extension modules. E.g. if numarray is used it is often difficult to estimate how much time is spent in the numarray extension and how much in application-specific Python code. Is it worth investing time reimplementing the application-specific code in C or C++? If most of the time is already spent in the numarray extension the answer will be no. Method: Determination of "micro-seconds per tick:" time.time() / sys.gettickeraccumulation() * 1.e6 A "tick" is the processing of one "Python virtual instruction." See also "setcheckinterval" here: http://www.python.org/doc/current/lib/module-sys.html A pure Python program will spent the bulk of the time interpreting bytecode. This will lead to a small value for "micro-seconds per tick." In contrast, if almost all the time is spent in extension modules, a calculation may run a long time without increasing sys.gettickeraccumulation(). For example: time ticks time/tick in micro-seconds 0.00% Python 0.008 20 405.908 10.00% Python 0.072 444680 0.163 20.00% Python 0.138 889340 0.155 30.00% Python 0.203 1334000 0.152 ... 100.00% Python 0.659 4446620 0.148 With higher resolution: time ticks time/tick in micro-seconds 0.00% Python 6.758 20 337893.748 0.10% Python 0.728 444680 1.638 0.20% Python 0.795 889340 0.894 0.30% Python 0.837 1334000 0.627 0.40% Python 0.923 1778660 0.519 0.50% Python 0.978 2223320 0.440 0.60% Python 1.091 2667980 0.409 0.70% Python 1.111 3112640 0.357 0.80% Python 1.191 3557300 0.335 0.90% Python 1.236 4001960 0.309 1.00% Python 1.315 4446620 0.296 In real applications we observed times/tick of around 10 on the same platform, indicating that significant performance increases could only be achieved through extensive recoding in a compiled language. On the other hand if we see values smaller than 1 we know that significant performance increases are achievable with a reasonable effort. Implementation: A full patch based on Python-2.4b2 is attached. The essence is this additional line in Python/ceval.c: + _Py_TickerAccumulation += _Py_CheckInterval - _Py_Ticker; Note that _Py_Ticker and _Py_CheckInterval exist already in the Python distribution. The impact of the additional code on the runtime performance of a pure Python program is minute. On Xeon/Linux the factor is smaller than 1.00005. The factor is even smaller if extension modules are used. The full patch, the patched files, and a complete Python distribution including the patched files can be found here: http://cci.lbl.gov/~rwgk/python/Python-2.4b2_ticker_patch http://cci.lbl.gov/~rwgk/python/Python-2.4b2_ticker_patched_files.tar.gz http://cci.lbl.gov/~rwgk/python/Python-2.4b2_ticker.tar.gz diff -u -r Python-2.4b2/Include/ceval.h Python-2.4b2_ticker/Include/ceval.h --- Python-2.4b2/Include/ceval.h 2004-10-10 19:40:35.000000000 -0700 +++ Python-2.4b2_ticker/Include/ceval.h 2004-11-12 18:16:28.000000000 -0800 @@ -68,6 +68,11 @@ /* this used to be handled on a per-thread basis - now just two globals */ PyAPI_DATA(volatile int) _Py_Ticker; +#ifndef HAVE_LONG_LONG +PyAPI_DATA(volatile long) _Py_TickerAccumulation; +#else +PyAPI_DATA(volatile PY_LONG_LONG) _Py_TickerAccumulation; +#endif PyAPI_DATA(int) _Py_CheckInterval; /* Interface for threads. diff -u -r Python-2.4b2/Objects/longobject.c Python-2.4b2_ticker/Objects/longobject.c --- Python-2.4b2/Objects/longobject.c 2004-09-19 23:14:54.000000000 -0700 +++ Python-2.4b2_ticker/Objects/longobject.c 2004-11-12 18:20:53.000000000 -0800 @@ -38,6 +38,7 @@ #define SIGCHECK(PyTryBlock) \ if (--_Py_Ticker < 0) { \ + _Py_TickerAccumulation += _Py_CheckInterval - _Py_Ticker; \ _Py_Ticker = _Py_CheckInterval; \ if (PyErr_CheckSignals()) { PyTryBlock; } \ } diff -u -r Python-2.4b2/PC/os2emx/python24.def Python-2.4b2_ticker/PC/os2emx/python24.def --- Python-2.4b2/PC/os2emx/python24.def 2004-10-10 19:40:50.000000000 -0700 +++ Python-2.4b2_ticker/PC/os2emx/python24.def 2004-11-12 18:16:47.000000000 -0800 @@ -743,6 +743,7 @@ "_Py_CheckRecursionLimit" "_Py_CheckInterval" "_Py_Ticker" + "_Py_TickerAccumulation" ; From python24_s.lib(compile) "PyCode_New" diff -u -r Python-2.4b2/Python/ceval.c Python-2.4b2_ticker/Python/ceval.c --- Python-2.4b2/Python/ceval.c 2004-10-10 19:40:50.000000000 -0700 +++ Python-2.4b2_ticker/Python/ceval.c 2004-11-12 18:22:55.000000000 -0800 @@ -373,6 +373,7 @@ pendinglast = j; _Py_Ticker = 0; + _Py_TickerAccumulation = 0; things_to_do = 1; /* Signal main loop */ busy = 0; /* XXX End critical section */ @@ -476,6 +477,11 @@ per thread, now just a pair o' globals */ int _Py_CheckInterval = 100; volatile int _Py_Ticker = 100; +#ifndef HAVE_LONG_LONG +volatile long _Py_TickerAccumulation = 0; +#else +volatile PY_LONG_LONG _Py_TickerAccumulation = 0; +#endif PyObject * PyEval_EvalCode(PyCodeObject *co, PyObject *globals, PyObject *locals) @@ -776,6 +782,7 @@ a try: finally: block uninterruptable. */ goto fast_next_opcode; } + _Py_TickerAccumulation += _Py_CheckInterval - _Py_Ticker; _Py_Ticker = _Py_CheckInterval; tstate->tick_counter++; #ifdef WITH_TSC diff -u -r Python-2.4b2/Python/sysmodule.c Python-2.4b2_ticker/Python/sysmodule.c --- Python-2.4b2/Python/sysmodule.c 2004-08-12 11:19:17.000000000 -0700 +++ Python-2.4b2_ticker/Python/sysmodule.c 2004-11-12 18:51:14.000000000 -0800 @@ -442,6 +442,20 @@ "getcheckinterval() -> current check interval; see setcheckinterval()." ); +static PyObject * +sys_gettickeraccumulation(PyObject *self, PyObject *args) +{ +#ifndef HAVE_LONG_LONG + return PyInt_FromLong(_Py_TickerAccumulation + _Py_CheckInterval - _Py_Ticker); +#else + return PyLong_FromLongLong(_Py_TickerAccumulation + _Py_CheckInterval - _Py_Ticker); +#endif +} + +PyDoc_STRVAR(gettickeraccumulation_doc, +"gettickeraccumulation() -> current count of bytecodes processed by the interpreter." +); + #ifdef WITH_TSC static PyObject * sys_settscdump(PyObject *self, PyObject *args) @@ -763,6 +777,8 @@ setcheckinterval_doc}, {"getcheckinterval", sys_getcheckinterval, METH_NOARGS, getcheckinterval_doc}, + {"gettickeraccumulation", sys_gettickeraccumulation, METH_NOARGS, + gettickeraccumulation_doc}, #ifdef HAVE_DLOPEN {"setdlopenflags", sys_setdlopenflags, METH_VARARGS, setdlopenflags_doc}, From anthony at interlink.com.au Sat Nov 13 05:34:20 2004 From: anthony at interlink.com.au (Anthony Baxter) Date: Sat Nov 13 05:34:44 2004 Subject: [Python-Dev] proposal+patch: sys.gettickeraccumulation() In-Reply-To: <200411130420.iAD4K3K2006945@boa.lbl.gov> References: <200411130420.iAD4K3K2006945@boa.lbl.gov> Message-ID: <41958ECC.7080100@interlink.com.au> Ralf W. Grosse-Kunstleve wrote: > Proposal: > > >>> sys.gettickeraccumulation.__doc__ > 'getgettickeraccumulation() -> current count of bytecodes processed by the interpreter.' This doesn't offer enough to be worth rolling into the 2.4 code. Anthony -- Anthony Baxter <anthony@interlink.com.au> It's never too late to have a happy childhood. From johahn at home.se Thu Nov 11 22:50:21 2004 From: johahn at home.se (Johan Hahn) Date: Sat Nov 13 05:35:40 2004 Subject: [Python-Dev] syntactic shortcut - unpack to variably sized list Message-ID: <000c01c4c838$77bb3da0$0d00a8c0@MAINFRAME> Hi As far as I can tell from the archive, this has not been discussed before. This is the second time in less than a week that I have stumbled over the rather clumsy syntax of extracting some elements of a sequence and at the same time remove those from the sequence: >>> L = 'a b 1 2 3'.split(' ') >>> a,b,L = L[0], L[1], L[2:] I think it would be nice if the following was legal: >>> a,b,*L = 'a b 1 2 3'.split(' ') >>> a, b, L ('a', 'b', ['1', '2', '3']) Today, if the number of variables on both sides of the equal sign doesn't match, an exception is raised (for google reference): ValueError: unpack list of wrong size This new syntax is very similar to the special parameter in function definitions that catches all excess arguments, as in def func(p, --> *args <--, **kw):, and so the semantics should be what everyone expects. Then, if we leave this limiting analogy with the *args parameter and allow the catch-the-rest variable to be anywhere in the left-hand side, we arrive at a splice syntax that reminds me a little of how prolog deals with lists: >>> a,*b,c = 'a b 1 2 3'.split(' ') >>> *a,b,c = b.split(' ') >>> a,b,c (['b'], '1', '2') I believe this would make a nice addition to the language. (Please cc me in response as I'm not subscribed to py-dev.) ...johahn From anthony at interlink.com.au Sat Nov 13 05:35:30 2004 From: anthony at interlink.com.au (Anthony Baxter) Date: Sat Nov 13 05:35:42 2004 Subject: [Python-Dev] struct module: add OverflowErrors now? In-Reply-To: <20041112131236.GA29540@vicky.ecs.soton.ac.uk> References: <20041112131236.GA29540@vicky.ecs.soton.ac.uk> Message-ID: <41958F12.7040406@interlink.com.au> Armin Rigo wrote: > Hi, > > The struct module is too lazy in checking for overflows (it only checks some > cases on some architectures). See patch #1038854. The question is: can we > make it strict this late in the 2.4 release process? This *will* break some > user code; the proof is that it breaks some standard library code, e.g. > gzip.py, which uses a signed typecode when it really means an unsigned one. > > Fixing it in 2.4 would be nice, but on the other hand we could postpone it for > the 2.5 trunk; people trying 2.5 early will see the problem quickly. We could > add warnings for 2.4 but that looks overkill. Or we could just go ahead and > break existing buggy code with 2.4b3. Opinions? I think I'd prefer to delay it until 2.5. This is going to break a _lot_ of code. Adding DeprecationWarnings for 2.4 would be more appropriate, imho. -- Anthony Baxter <anthony@interlink.com.au> It's never too late to have a happy childhood. From foom at fuhm.net Sat Nov 13 08:02:23 2004 From: foom at fuhm.net (James Y Knight) Date: Sat Nov 13 08:02:37 2004 Subject: [Python-Dev] syntactic shortcut - unpack to variably sized list In-Reply-To: <000c01c4c838$77bb3da0$0d00a8c0@MAINFRAME> References: <000c01c4c838$77bb3da0$0d00a8c0@MAINFRAME> Message-ID: <F80C7DCC-3541-11D9-AA57-000A95A50FB2@fuhm.net> On Nov 11, 2004, at 4:50 PM, Johan Hahn wrote: > As far as I can tell from the archive, this has not been discussed > before. > [...] > I think it would be nice if the following was legal: > >>> a,b,*L = 'a b 1 2 3'.split(' ') > >>> a, b, L > ('a', 'b', ['1', '2', '3']) Oh, this has been discussed before. It's come up quite a number of times, in fact. But, it's been rejected every time. FWIW [which is very little ;)], I agree: this would be nice. I'm strongly for consistency and orthogonality: there should be few distinct concepts in the language each of which works in many places. Allowing * only in an function call but not in other tuple-constructing situations violates this concept, IMO. It would have been great if this had been added at the same time as * in function calls. Adding it now may be more trouble than it's worth, though. Two threads on the topic, with rejections by Guido: http://mail.python.org/pipermail/python-dev/2002-November/030349.html http://mail.python.org/pipermail/python-dev/2004-August/046684.html Guido's rejection from http://mail.python.org/pipermail/python-dev/2004-August/046794.html: > For the record, let me just say that I'm -1 on adding this feature > now. It's only a minor convenience while it's a fairly big change to > the parser and code generator, but most of all, Python's growth needs > to be reduced. If we were to add every "nice" feature that was > invented (or present in other languages), Python would lose one of its > main charms, which is that it is really a rather simple language that > can be learned and put to production quickly. > > So while I appreciate the idea (which BTW has been proposed before) I > think it's not worth adding now, and if you don't hear from me again > on this topic it's because I haven't changed my mind... James From rwgk at cci.lbl.gov Sat Nov 13 10:19:19 2004 From: rwgk at cci.lbl.gov (Ralf W. Grosse-Kunstleve) Date: Sat Nov 13 10:19:28 2004 Subject: [Python-Dev] proposal+patch: sys.gettickeraccumulation() Message-ID: <200411130919.iAD9JJxP009427@boa.lbl.gov> > > Proposal: > > > > >>> sys.gettickeraccumulation.__doc__ > > 'getgettickeraccumulation() -> current count of bytecodes processed by the interpreter.' > > This doesn't offer enough to be worth rolling into the 2.4 code. I could easily provide a similar patch based on the mainline CVS. Ralf From anthony at interlink.com.au Sat Nov 13 11:52:54 2004 From: anthony at interlink.com.au (Anthony Baxter) Date: Sat Nov 13 11:53:08 2004 Subject: [Python-Dev] proposal+patch: sys.gettickeraccumulation() In-Reply-To: <200411130919.iAD9JJxP009427@boa.lbl.gov> References: <200411130919.iAD9JJxP009427@boa.lbl.gov> Message-ID: <4195E786.1090207@interlink.com.au> Ralf W. Grosse-Kunstleve wrote: >>This doesn't offer enough to be worth rolling into the 2.4 code. > > > I could easily provide a similar patch based on the mainline CVS. I'm not sure we're understanding each other here. What I was saying is that if this were to go into Python, it should go in after 2.4 final is out, and the release24-maint branch has been cut. Anthony -- Anthony Baxter <anthony@interlink.com.au> It's never too late to have a happy childhood. From martin at v.loewis.de Sat Nov 13 12:41:12 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat Nov 13 12:41:09 2004 Subject: [Python-Dev] syntactic shortcut - unpack to variably sized list In-Reply-To: <000c01c4c838$77bb3da0$0d00a8c0@MAINFRAME> References: <000c01c4c838$77bb3da0$0d00a8c0@MAINFRAME> Message-ID: <4195F2D8.6010208@v.loewis.de> Johan Hahn wrote: > Hi > > As far as I can tell from the archive, this has not been discussed before. > This is the second time in less than a week that I have stumbled over the rather > clumsy syntax of extracting some elements of a sequence and at the same time > remove those from the sequence: > >>>>L = 'a b 1 2 3'.split(' ') >>>>a,b,L = L[0], L[1], L[2:] As James says, this has been discussed before. Typically, you don't need the rest list, so you write a,b = 'a b 1 2 3'.split(' ')[:2] If you do need the rest list, it might be best to write L = 'a b 1 2 3'.split(' ') a = L.pop(0) b = L.pop(0) Whether this is more efficient than creating a new list depends on the size of the list; so you might want to write L = 'a b 1 2 3'.split(' ') a = L[0] b = L[1] del L[:2] This is entirely different from functions calls, which you simply cannot spread over several statements. Regards, Martin From martin at v.loewis.de Sat Nov 13 12:48:15 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat Nov 13 12:48:16 2004 Subject: [Python-Dev] [development doc updates] In-Reply-To: <20041113064452.B36C518E89A@grendel.fdrake.net> References: <20041113064452.B36C518E89A@grendel.fdrake.net> Message-ID: <4195F47F.5050504@v.loewis.de> Fred L. Drake wrote: > These changes are experimental at this point, and may be changed > further or abandoned. I'm very interested in getting feedback on > these changes; if people like them, they can be put into place for > Python 2.4. I like the changes. > Please direct feedback to the Python Documentation SIG mailing list: I would have, but my mail to this list was rejected. Regards, Martin From ncoghlan at iinet.net.au Sat Nov 13 13:05:23 2004 From: ncoghlan at iinet.net.au (Nick Coghlan) Date: Sat Nov 13 13:05:30 2004 Subject: [Python-Dev] syntactic shortcut - unpack to variably sized list In-Reply-To: <000c01c4c838$77bb3da0$0d00a8c0@MAINFRAME> References: <000c01c4c838$77bb3da0$0d00a8c0@MAINFRAME> Message-ID: <4195F883.8030309@iinet.net.au> Johan Hahn wrote: > Hi > > As far as I can tell from the archive, this has not been discussed before. > This is the second time in less than a week that I have stumbled over the rather > clumsy syntax of extracting some elements of a sequence and at the same time > remove those from the sequence: > >>>>L = 'a b 1 2 3'.split(' ') >>>>a,b,L = L[0], L[1], L[2:] > > > I think it would be nice if the following was legal: > >>>>a,b,*L = 'a b 1 2 3'.split(' ') >>>>a, b, L > > ('a', 'b', ['1', '2', '3']) Hmm - I just had a thought about this. Is it worth adding a "len" argument to list.pop? (The idea was inspired by Martin's use of list.pop to handle the above case). With that change, the above example would become: a, b = L.pop(0, 2) At the moment, list.pop is described as equivalent to: x = L[i]; del L[i]; return x with this change, it would be: x = L[i:i+n]; del L[i:i+n]; return x By default, n = 1, so the standard behaviour of list.pop is preserved. Cheers, Nick. -- Nick Coghlan | Brisbane, Australia Email: ncoghlan@email.com | Mobile: +61 409 573 268 From martin at v.loewis.de Sat Nov 13 14:03:15 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat Nov 13 14:03:11 2004 Subject: [Python-Dev] syntactic shortcut - unpack to variably sized list In-Reply-To: <4195F883.8030309@iinet.net.au> References: <000c01c4c838$77bb3da0$0d00a8c0@MAINFRAME> <4195F883.8030309@iinet.net.au> Message-ID: <41960613.1000307@v.loewis.de> Nick Coghlan wrote: > x = L[i:i+n]; del L[i:i+n]; return x > > By default, n = 1, so the standard behaviour of list.pop is preserved. This default would actually change the standard behaviour: whereas it now returns a single element, it would then return a list containing the single element. Regards, Martin From johahn2003 at home.se Sat Nov 13 14:16:41 2004 From: johahn2003 at home.se (Johan Hahn) Date: Sat Nov 13 14:20:07 2004 Subject: [Python-Dev] syntactic shortcut - unpack to variably sized list References: <000c01c4c838$77bb3da0$0d00a8c0@MAINFRAME><4195F883.8030309@iinet.net.au> <41960613.1000307@v.loewis.de> Message-ID: <033201c4c983$045a1dd0$0d00a8c0@MAINFRAME> Martin v. L?wis wrote: > Nick Coghlan wrote: > > x = L[i:i+n]; del L[i:i+n]; return x > > > > By default, n = 1, so the standard behaviour of list.pop is preserved. > > This default would actually change the standard behaviour: whereas it > now returns a single element, it would then return a list containing > the single element. Too bad. I would have liked list.pop(i, n=1, step=1) to be the same as x = L[i:i+n:step]; del L[i:i+n:step]; return x Having pop()/pop(i) return an element and pop(i, n)/pop(i, n, step) return a list is a no-no, or is it? ...johahn From ncoghlan at iinet.net.au Sat Nov 13 14:41:25 2004 From: ncoghlan at iinet.net.au (Nick Coghlan) Date: Sat Nov 13 14:41:33 2004 Subject: [Python-Dev] syntactic shortcut - unpack to variably sized list In-Reply-To: <41960613.1000307@v.loewis.de> References: <000c01c4c838$77bb3da0$0d00a8c0@MAINFRAME> <4195F883.8030309@iinet.net.au> <41960613.1000307@v.loewis.de> Message-ID: <41960F05.60006@iinet.net.au> Martin v. L?wis wrote: > Nick Coghlan wrote: > >> x = L[i:i+n]; del L[i:i+n]; return x >> >> By default, n = 1, so the standard behaviour of list.pop is preserved. > > > This default would actually change the standard behaviour: whereas it > now returns a single element, it would then return a list containing > the single element. Ah, good point. I'd see two possible fixes to that: a) have n=0 be the default, and mean 'give me the element, not a list with 1 element" (L.pop(0,1) would then mean "give me a list containing only the first element"). That's a little magical for my taste, though. b) have a new method called 'extract' or 'poplist' or 'popslice' or similar (with the behaviour given above) Actually, if we went with the b) option, and the name 'popslice', I would suggest the following signature: list.popslice(start=0, end, step=1) i.e. L.popslice(a, b, c) is to L[a:b:c] as L.pop(a) is to L[a] And, returning once again to the OP's example, we would have: a, b = list.popslice(2) Cheers, Nick. -- Nick Coghlan | Brisbane, Australia Email: ncoghlan@email.com | Mobile: +61 409 573 268 From ncoghlan at iinet.net.au Sat Nov 13 14:50:30 2004 From: ncoghlan at iinet.net.au (Nick Coghlan) Date: Sat Nov 13 14:50:36 2004 Subject: [Python-Dev] syntactic shortcut - unpack to variably sized list In-Reply-To: <41960F05.60006@iinet.net.au> References: <000c01c4c838$77bb3da0$0d00a8c0@MAINFRAME> <4195F883.8030309@iinet.net.au> <41960613.1000307@v.loewis.de> <41960F05.60006@iinet.net.au> Message-ID: <41961126.7060101@iinet.net.au> Nick Coghlan wrote: > Actually, if we went with the b) option, and the name 'popslice', I > would suggest the following signature: > > list.popslice(start=0, end, step=1) "Single argument is end value" was modelled on range(), FWIW. Cheers, Nick. -- Nick Coghlan | Brisbane, Australia Email: ncoghlan@email.com | Mobile: +61 409 573 268 From ncoghlan at iinet.net.au Sat Nov 13 15:04:44 2004 From: ncoghlan at iinet.net.au (Nick Coghlan) Date: Sat Nov 13 15:04:51 2004 Subject: [Python-Dev] syntactic shortcut - unpack to variably sized list In-Reply-To: <41961126.7060101@iinet.net.au> References: <000c01c4c838$77bb3da0$0d00a8c0@MAINFRAME> <4195F883.8030309@iinet.net.au> <41960613.1000307@v.loewis.de> <41960F05.60006@iinet.net.au> <41961126.7060101@iinet.net.au> Message-ID: <4196147C.8050303@iinet.net.au> Nick Coghlan wrote: > Nick Coghlan wrote: > >> Actually, if we went with the b) option, and the name 'popslice', I >> would suggest the following signature: >> >> list.popslice(start=0, end, step=1) > > > "Single argument is end value" was modelled on range(), FWIW. Heh - turns out there is an actual slice() constructor with that start/end/step signature (I've never used it, so I didn't even know it was there). Anyway, *if* we do anything about this (cleaner unpacking & removal of part of a list), I think 'popslice' or something similar is the right way to address it (rather than monkeying with syntax, which would only address slices at the start of the list anyway). Any of the "a, b, *c = L" proponents want to come up with a candidate 'list.popslice' patch for 2.5? The implementation shouldn't be too hard - the "x = L[a:b:c]; del L[a:b:c]; return x" description should suffice for an implementation, too (using the C API, rather than actual Python, though). Cheers, Nick. -- Nick Coghlan | Brisbane, Australia Email: ncoghlan@email.com | Mobile: +61 409 573 268 From jjl at pobox.com Sat Nov 13 15:37:53 2004 From: jjl at pobox.com (John J Lee) Date: Sat Nov 13 15:38:36 2004 Subject: [Python-Dev] Minor cookielib changes for 2.4b1 In-Reply-To: <002601c4b2de$98ed7840$e841fea9@oemcomputer> References: <002601c4b2de$98ed7840$e841fea9@oemcomputer> Message-ID: <Pine.LNX.4.58.0411131431010.6308@alice> Sorry for not replying earlier: I've not been well. On Fri, 15 Oct 2004, Raymond Hettinger wrote: > > John J Lee wrote: > > > The patch contains uncontroversial changes to cookielib and > associated > > > modules, documentation and tests. Would be unfortunate not to have > these > > > in 2.4.0. > > [Martin] > > What precisely makes these changes uncontroversial? The fact that no > > controversion has been developed? My attempt at time-travel was unsuccessful, sorry. :-) > I've looked over the patch and think most of it should wait until Py2.5. > Refactorings of this magnitude should be done early in the development > cycle to allow time to discover and fix issues if something gets mucked > up. Hadn't realised how late it was. > A couple of the doc changes should probably go in for Py2.4b2 (a note on > thread safety and an example of how to display debugging information). > If no one objects, I'll put those in after the beta. Also there is a > minor readability fix (using escape sequences) for the test suite that > is reasonable for Py2.4. Thanks, Raymond. John From rwgk at cci.lbl.gov Sat Nov 13 18:16:05 2004 From: rwgk at cci.lbl.gov (Ralf W. Grosse-Kunstleve) Date: Sat Nov 13 18:16:15 2004 Subject: [Python-Dev] proposal+patch: sys.gettickeraccumulation() Message-ID: <200411131716.iADHG59n012286@boa.lbl.gov> > it should go in after 2.4 > final is out, and the release24-maint branch has been cut. OK, I'll wait until 2.4 is out and submit another patch based on the mainline CVS through the SourceForge system. > if this were to go into Python - What speaks against making the ticker accumulation available at the script level? - Is there an alternative way of achieving the goal of my proposal? It was: > A fast, easy, and non-intrusive method for estimating how much time > is spent in Python code and how much time in extension modules. Thanks! Ralf From ncoghlan at iinet.net.au Sun Nov 14 10:45:42 2004 From: ncoghlan at iinet.net.au (Nick Coghlan) Date: Sun Nov 14 10:45:50 2004 Subject: [Python-Dev] syntactic shortcut - unpack to variably sized list In-Reply-To: <41964CF2.2090209@botanicus.net> References: <000c01c4c838$77bb3da0$0d00a8c0@MAINFRAME> <4195F883.8030309@iinet.net.au> <41960613.1000307@v.loewis.de> <41960F05.60006@iinet.net.au> <41964CF2.2090209@botanicus.net> Message-ID: <41972946.8070207@iinet.net.au> David Wilson wrote: > I dislike the use of 'popslice' as it implies somewhere along the lines > the use of the slice type, which isn't happening. If I was writing it, using a slice object as an argument to PySequence_GetItem is *exactly* the method I would be using to implement it: indices = PySlice_New(start, end, step); x = PySequence_GetItem(L, indices); PySequence_DelItem(L, indices); return x; Add reference counting, error checking and actually hook this up to the list type in the interpreter, and you're basically done. Heck, it could be argued that list.pop should just accept slices for the argument: a, b = L.pop(slice(2)) # Pop the first two items x, y, z = L.pop(slice(1:6:2)) # Pop items 1, 3 & 5 No new method required in that case - if you provide an index to list.pop, it will return a single item, if you provide a slice object, it will return a list. (Similar to the way list.__getitem__ currently works) > Also, what if at some > future date it is decided that all magical syntax should be exposed > using friendly method names too? Then you might have a list.slice which > work in a much different way.[1] List slicing is already exposed in Python in a few ways: L[:2] L.__getitem__(slice(2)) operator.itemgetter(slice(2))(L) Adding yet another way to spell it doesn't seem too likely. Given that lists *had* a __getslice__ method, and it has since been deprecated in favour of slice() objects, I'd be especially surprised to see list.slice (or equivalent) making a comeback. I will note that I only looked into the history of slicing this evening (when checking the C API for the sample implementation above). Having done so, the trend seems to have been away from having special methods for handling slices (__getslice__, __setslice__, __delslice__), and towards methods which can handle either a numeric index or a slice object (__getitem__, __setitem__, __delitem__). So maybe "list.pop(slice(2))" is the way to go (a non-optimised version could be written fairly easily using the existing internal list functions to retrieve and then delete the data. The main trick would be to leave the standard path through listpop alone to avoid any performance hit to the current users of list.pop() and list.pop(x)). > Second of all, you cannot have a non-default argument follow a default > argument, so your proposed syntax would not work without breaking > standard semantics and writing special code to act differently depending > on the argument list length, and also take into account if keyword args > are used, etc, etc. The signature I proposed is identical to the existing signatures for the builtins range(), xrange() and slice(). It's true that normal Python functions and methods can't have a signature like that (it's a syntax error). C extensions (including the CPython interpreter) can do whatever they want, though. In this case, I think being consistent with slice() is more important than being consistent with normal function signatures. However, the suggestion above (allowing slice objects as arguments to list.pop) takes advantage of the existing slice() builtin, so none of that non-standard argument handling would need to be duplicated. > For an extra method, +1, but with a different name (I was thinking > 'popitems', but dict.popitem returns a (k, v)). What do you think of allowing slice objects to be used with list.pop()? > Oh one last thought. If "*<identifier>" were to be added, then what of > "**<identifier>"? I can't think of any use for it, and so the whole > scheme might appear confusing to beginners who find they can use ** in > function declarations, but not in expressions. Aye, I don't think it's an accident that "a, b *c = L" has been knocked on the head 3 times now. My main gripe with it is that it requires new syntax, yet still only supports a small subset of slicing. My idea about enhancing list.pop or adding a new method to list to provide the desired functionality is aimed at avoiding having the discussion *again* in the future (since clearly this is functionality some portion of the user community would make use of). I don't actually need the functionality myself, so I doubt I'll get around to providing a sample implementation. But the suggestion is there if any of the people that *do* want the feature would like to implement it. > [1] Exposing magical syntax (eg. indexers) as real methods is the > Microsoft way for .NET (or at least, according to a C# book I read). It > proves useful in occasions where interoperability with lesser systems is > required, for example, a language that does not support custom indexers, > or for enabling clients of an object bridge [that does not support > cust..] to make full use of your classes. Python already exposes most things as methods - I found browsing through the Language Reference a few months back extremely enlightening in that regard (specifically Section 3.3 Special Method Names). Cheers, Nick. -- Nick Coghlan | Brisbane, Australia Email: ncoghlan@email.com | Mobile: +61 409 573 268 From mwh at python.net Sun Nov 14 12:08:37 2004 From: mwh at python.net (Michael Hudson) Date: Sun Nov 14 12:08:39 2004 Subject: [Python-Dev] proposal+patch: sys.gettickeraccumulation() In-Reply-To: <200411130420.iAD4K3K2006945@boa.lbl.gov> (Ralf W. Grosse-Kunstleve's message of "Fri, 12 Nov 2004 20:20:03 -0800 (PST)") References: <200411130420.iAD4K3K2006945@boa.lbl.gov> Message-ID: <2m3bzc1ydm.fsf@starship.python.net> "Ralf W. Grosse-Kunstleve" <rwgk@cci.lbl.gov> writes: > Proposal: > > >>> sys.gettickeraccumulation.__doc__ > 'getgettickeraccumulation() -> current count of bytecodes processed by the interpreter.' > You know about the 'tick_counter' field of PyThreadState? I don't think it's accessible to Python currently, but it seems to be intended for similar purposes... Cheers, mwh -- Hiro dicks about with his computer, naturally. Being stranded on a life raft in the Pacific is a perfect venue for a hacker. -- Snow Crash, Neal Stephenson From ncoghlan at gmail.com Sun Nov 14 13:30:03 2004 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun Nov 14 13:30:15 2004 Subject: [Python-Dev] proposal+patch: sys.gettickeraccumulation() In-Reply-To: <200411130420.iAD4K3K2006945@boa.lbl.gov> References: <200411130420.iAD4K3K2006945@boa.lbl.gov> Message-ID: <41974FCB.4090002@iinet.net.au> Ralf W. Grosse-Kunstleve wrote: > A pure Python program will spent the bulk of the time interpreting > bytecode. Perhaps, perhaps not. A heck of a lot of the Python core isn't written in Python - time spent executing builtins, or running methods of builtin objects usually doesn't involve the main interpreter loop (we're generally in pure C-code at that point). I'm curious how the suggested feature can provide any information that is actually useful for optimisation purposes. Just because a low proportion of time is spent in Python code, doesn't mean the Python code isn't at fault for poor performance. As an example, in CPython 2.3 and earlier, this: result = "" for x in strings: result += x is a lot worse performance-wise than: result = "".join(strings) The first version does spend more time in Python code, but the performance killer is actually in the string concatenation C code. So the time is spent in the C code, but the fault lies in the Python code (In Python 2.4, the latter version is still faster, but the difference isn't as dramatic as it used to be). Knowing "I'm spending x% of the time executing Python code" just isn't really all that interesting, unless you know *what* Python code is taking all the time. (e.g., if decimal.py spends all its time creating new instances of Decimal when running a benchmark, then you know to either speed up Decimal construction, or to look for temporary objects that can be eliminated, or both. Then you run the benchmark again to determine if the changes actually improved the situation) I'd rather encourage people to write appropriate benchmark scripts and execute them using "python -m profile <benchmark>", rather than lead them up the garden path with a global "Python/non-Python" percentage estimation utility. Cheers, Nick. -- Nick Coghlan | Brisbane, Australia Email: ncoghlan@email.com | Mobile: +61 409 573 268 From arigo at tunes.org Sun Nov 14 14:17:08 2004 From: arigo at tunes.org (Armin Rigo) Date: Sun Nov 14 14:29:37 2004 Subject: [Python-Dev] syntactic shortcut - unpack to variably sized list In-Reply-To: <4195F883.8030309@iinet.net.au> References: <000c01c4c838$77bb3da0$0d00a8c0@MAINFRAME> <4195F883.8030309@iinet.net.au> Message-ID: <20041114131708.GA9442@vicky.ecs.soton.ac.uk> Hello Nick, On Sat, Nov 13, 2004 at 10:05:23PM +1000, Nick Coghlan wrote: > At the moment, list.pop is described as equivalent to: > > x = L[i]; del L[i]; return x Then the documentation is not exact: if it were really equivalent to the above definition, then we could already do today: a, b = L.pop(slice(0,2)) because starting from Python 2.3, L[slice(0,2)] is equivalent to L[0:2] So who's wrong: the documentation of list.pop() or its implementation? It would be nicely regular if we could pop slices. Armin From rwgk at cci.lbl.gov Sun Nov 14 20:18:14 2004 From: rwgk at cci.lbl.gov (Ralf W. Grosse-Kunstleve) Date: Sun Nov 14 20:18:19 2004 Subject: [Python-Dev] proposal+patch: sys.gettickeraccumulation() Message-ID: <200411141918.iAEJIEvw023160@boa.lbl.gov> Nick Coghlan wrote: > Ralf W. Grosse-Kunstleve wrote: > > A pure Python program will spent the bulk of the time interpreting > > bytecode. > > Perhaps, perhaps not. Right. But remember what the actual goal is: we want to answer the question "Is it worth reimplementing a piece currently written in Python in C/C++?" > A heck of a lot of the Python core isn't written > in Python - time spent executing builtins, or running methods of builtin > objects usually doesn't involve the main interpreter loop (we're > generally in pure C-code at that point). If a piece of Python code leads to heavy use of complex, time-consuming builtin operations it will be of less benefit to reimplement that code in C/C++. This is exactly what we want to learn. > I'm curious how the suggested feature can provide any information that > is actually useful for optimisation purposes. Just because a low > proportion of time is spent in Python code, doesn't mean the Python code > isn't at fault for poor performance. > > As an example, in CPython 2.3 and earlier, this: > > result = "" > for x in strings: > result += x > > is a lot worse performance-wise than: > > result = "".join(strings) > > The first version does spend more time in Python code, but the > performance killer is actually in the string concatenation C code. So > the time is spent in the C code, but the fault lies in the Python code > (In Python 2.4, the latter version is still faster, but the difference > isn't as dramatic as it used to be). Exactly. If you try out my patch and look at time/ticks you will see immediately that there is no point in reimplementing "".join(strings) in C/C++. Importantly, you don't have to look at the code to arrive at this conclusion. The time/tick alone will tell you. This is very helpful if you are working with third-party code. > Knowing "I'm spending x% of the time executing Python code" just isn't > really all that interesting, Right. Sorry if I gave the wrong impression that this could be interesting. It is indeed not. What is interesting is the estimated benefit of reimplementing a piece of Python in C/C++. This is in fact highly correlated with the time/tick. > I'd rather encourage people to write appropriate benchmark scripts and > execute them using "python -m profile <benchmark> ", This approach is impractical/impossible in the real world. For example, this is the problem prompting me to implement sys.gettickeraccumulation(): http://pyquante.sourceforge.net/ - It is not our code, i.e. it is difficult for us to know where the time is spent. - It makes heavy use of Numeric. - It has a few innermost loops implemented in C. We are using only some parts of this library. Question: if we reimplement these parts completely in C++, what speedup can we expect? So we run a whole calculation and print the time/tick, which you can do with less than a one-minute investment: print time.time()/sys.gettickeraccumulation()*1.e6 as the last statement of your code. If the printed value is close 0.15 on our reference platform we know that the speedup will be in the neighborhood of 100. Any value higher than 0.15 indicates that the expected speedup will less. In our case the value was 0.35, and after we did the reimplementation in C++ we found a speedup of about 10. We have other applications with time/tick around 10. Just looking at this number tells us that there is not much to gain unless we completely eliminate Python. Bring in the cost for the C++ programmer and the increased cost of maintaining the C++ code compared to Python, and you know what we decided (not) to do. > rather than lead > them up the garden path with a global "Python/non-Python" percentage > estimation utility. Please consider that that utility is simply printing time.time()/sys.gettickeraccumulation(), that my patch is trivial, and that the runtime penalty is close to non-existing. Cheers, Ralf From martin at v.loewis.de Sun Nov 14 20:38:18 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun Nov 14 20:38:20 2004 Subject: [Python-Dev] syntactic shortcut - unpack to variably sized list In-Reply-To: <20041114131708.GA9442@vicky.ecs.soton.ac.uk> References: <000c01c4c838$77bb3da0$0d00a8c0@MAINFRAME> <4195F883.8030309@iinet.net.au> <20041114131708.GA9442@vicky.ecs.soton.ac.uk> Message-ID: <4197B42A.8020401@v.loewis.de> Armin Rigo wrote: > So who's wrong: the documentation of list.pop() or its implementation? > It would be nicely regular if we could pop slices. That's an interesting observation. I was somewhat concerned that this kind of overloading might be confusion, but given that it is even already documented (with nobody noticing), I'm now very much in favour of allowing to pop slices. Of course, whoever implements the change should make sure that slices are applicable consistently everwhere the documentation says they should be. Regards, Martin From kayschluehr at gmx.de Sun Nov 14 19:27:30 2004 From: kayschluehr at gmx.de (Kay Schluehr) Date: Sun Nov 14 21:30:24 2004 Subject: [Python-Dev] Int literals and method calls Message-ID: <4197A392.7080100@gmx.de> Hi , i wondered why string and bool literals have access methods and one can call simply >>> "1".__class__ <type 'str'> >>> False.__class__ <type 'bool'> But this won't be true for int literals and float literals except for those float literals that are terminated by a dot: >>> 1..__class__ <type 'float'> The expression below raises an error: >>> 1.__class__ Traceback ( File "<interactive input>", line 1 1.__class__ ^ SyntaxError: invalid syntax So it seems to be a parser-problem, related to the ambiguity of the terminating dot? Could this be patched? Thanks Kay From Scott.Daniels at Acm.Org Sun Nov 14 21:50:00 2004 From: Scott.Daniels at Acm.Org (Scott David Daniels) Date: Sun Nov 14 21:48:44 2004 Subject: [Python-Dev] Re: Int literals and method calls In-Reply-To: <4197A392.7080100@gmx.de> References: <4197A392.7080100@gmx.de> Message-ID: <cn8gb5$cm0$1@sea.gmane.org> Kay Schluehr wrote: > But this won't be true for int literals and float literals ... > The expression below raises an error: > > >>> 1.__class__ > Traceback ( File "<interactive input>", line 1 > 1.__class__ > ^ > SyntaxError: invalid syntax But the expressions below both work: >>> 2 .__class__ <type 'int'> >>> (3).__class__ <type 'int'> > So it seems to be a parser-problem, To the extent it _is_ a problem, it is a lexer problem. However, "In the face of ambiguity, refuse to guess." -- Scott David Daniels Scott.Daniels@Acm.Org From martin at v.loewis.de Sun Nov 14 22:30:49 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun Nov 14 22:30:52 2004 Subject: [Python-Dev] Int literals and method calls In-Reply-To: <4197A392.7080100@gmx.de> References: <4197A392.7080100@gmx.de> Message-ID: <4197CE89.2060801@v.loewis.de> Kay Schluehr wrote: > >>> 1.__class__ > Traceback ( File "<interactive input>", line 1 > 1.__class__ > ^ > SyntaxError: invalid syntax > > So it seems to be a parser-problem, related to the ambiguity of the > terminating dot? There is no ambiguity. This expression parses as two tokens, floatnumber (through the second alternative of pointfloat), followed by identifier. While this is not explicitly specified in the language reference, Python follows the common "longest match" approach to tokenization, i.e. you extend each token from the beginning of the input as much as you can, and then proceed to the next token. > Could this be patched? Perhaps. But it should not, as the current behaviour is intentional. If you want to take attributeref of an integer literal, use one of the following forms: >>> 2 .__class__ <type 'int'> >>> (3).__class__ <type 'int'> Regards, Martin From mike at skew.org Sun Nov 14 22:49:29 2004 From: mike at skew.org (Mike Brown) Date: Sun Nov 14 22:49:32 2004 Subject: [Python-Dev] parse xml In-Reply-To: <46D2E28D-34F9-11D9-93D6-0003938E6950@ce.anoka.k12.mn.us> "from Mike Tuller at Nov 12, 2004 04:22:02 pm" Message-ID: <200411142149.iAELnTIr006016@chilled.skew.org> Mike Tuller wrote: > I am just starting with python. I have a problem where I want to take a > document that is in xml [...] You may get any number of helpful answers (zero is a number too, heh) if you post this to the Python XML SIG list. There are various ways to go about it, both with Python's built-in XML parsing capabilities, and with external packages. See http://mail.python.org/mailman/listinfo/xml-sig/ From greg.ewing at canterbury.ac.nz Mon Nov 15 05:12:50 2004 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon Nov 15 05:12:59 2004 Subject: [Python-Dev] Int literals and method calls In-Reply-To: <4197A392.7080100@gmx.de> References: <4197A392.7080100@gmx.de> Message-ID: <41982CC2.80006@canterbury.ac.nz> Kay Schluehr wrote: 1.__class__ > ^ > SyntaxError: invalid syntax > > Could this be patched? Why bother? Surely it's not too much trouble to type (1).__class__ on the rare occasions when you want to do something like this. -- Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From michael.walter at gmail.com Mon Nov 15 05:44:24 2004 From: michael.walter at gmail.com (Michael Walter) Date: Mon Nov 15 05:44:27 2004 Subject: [Python-Dev] Int literals and method calls In-Reply-To: <41982CC2.80006@canterbury.ac.nz> References: <4197A392.7080100@gmx.de> <41982CC2.80006@canterbury.ac.nz> Message-ID: <877e9a1704111420441c9e683c@mail.gmail.com> It's confusing/inconsistent. On Mon, 15 Nov 2004 17:12:50 +1300, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote: > Kay Schluehr wrote: > 1.__class__ > > ^ > > SyntaxError: invalid syntax > > > > Could this be patched? > > Why bother? Surely it's not too much trouble to type > > (1).__class__ > > on the rare occasions when you want to do something > like this. > > -- > Greg Ewing, Computer Science Dept, +--------------------------------------+ > University of Canterbury, | A citizen of NewZealandCorp, a | > Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | > greg@cosc.canterbury.ac.nz +--------------------------------------+ > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/michael.walter%40gmail.com > From martin at v.loewis.de Mon Nov 15 07:16:19 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon Nov 15 07:16:21 2004 Subject: [Python-Dev] Int literals and method calls In-Reply-To: <877e9a1704111420441c9e683c@mail.gmail.com> References: <4197A392.7080100@gmx.de> <41982CC2.80006@canterbury.ac.nz> <877e9a1704111420441c9e683c@mail.gmail.com> Message-ID: <419849B3.9020406@v.loewis.de> Michael Walter wrote: > It's confusing/inconsistent. That is not true. It might be confusing, but it is not inconsistent. Regards, Martin From michael.walter at gmail.com Mon Nov 15 08:30:15 2004 From: michael.walter at gmail.com (Michael Walter) Date: Mon Nov 15 08:30:18 2004 Subject: [Python-Dev] Int literals and method calls In-Reply-To: <419849B3.9020406@v.loewis.de> References: <4197A392.7080100@gmx.de> <41982CC2.80006@canterbury.ac.nz> <877e9a1704111420441c9e683c@mail.gmail.com> <419849B3.9020406@v.loewis.de> Message-ID: <877e9a170411142330661d3a33@mail.gmail.com> I think it's inconsistent because it works for "list literals" but not for "integer literals". What do I miss? Cheers, Michael On Mon, 15 Nov 2004 07:16:19 +0100, Martin v. L?wis <martin@v.loewis.de> wrote: > Michael Walter wrote: > > It's confusing/inconsistent. > > That is not true. It might be confusing, but it is not inconsistent. > > Regards, > Martin > From gerrit at nl.linux.org Mon Nov 15 10:54:01 2004 From: gerrit at nl.linux.org (Gerrit) Date: Mon Nov 15 10:54:28 2004 Subject: [Python-Dev] Int literals and method calls In-Reply-To: <419849B3.9020406@v.loewis.de> References: <4197A392.7080100@gmx.de> <41982CC2.80006@canterbury.ac.nz> <877e9a1704111420441c9e683c@mail.gmail.com> <419849B3.9020406@v.loewis.de> Message-ID: <20041115095401.GA19806@nl.linux.org> "Martin v. L?wis" wrote: > Michael Walter wrote: > >It's confusing/inconsistent. > > That is not true. It might be confusing, but it is not inconsistent. It works for complex numbers: >>> 1j.imag 1.0 ...in a rather surprising way: >>> 3+4j.real 3.0 >>> f() + 5j.real 42.0 I would have expected it to be 0.0, because I would have expected the '.' to have precedence over the '+', but it hasn't. apparantly. Or has it? >>> 3+4j.__class__ Traceback (most recent call last): File "<stdin>", line 1, in ? TypeError: unsupported operand type(s) for +: 'int' and 'type' I would call it inconsistent: 3+4j.real means (3+4j).real 3+4j.__class__ means 4+(4j).__class__ Enough ignorance from my side? :-) regards, Gerrit Holl. -- Weather in Twenthe, Netherlands 15/11 09:25: 5.0?C Few clouds overcast wind 3.1 m/s SSW (57 m above NAP) -- In the councils of government, we must guard against the acquisition of unwarranted influence, whether sought or unsought, by the military-industrial complex. The potential for the disastrous rise of misplaced power exists and will persist. -Dwight David Eisenhower, January 17, 1961 From ncoghlan at gmail.com Mon Nov 15 11:44:15 2004 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon Nov 15 11:44:25 2004 Subject: [Python-Dev] proposal+patch: sys.gettickeraccumulation() In-Reply-To: <200411141918.iAEJIEvw023160@boa.lbl.gov> References: <200411141918.iAEJIEvw023160@boa.lbl.gov> Message-ID: <4198887F.8000100@iinet.net.au> Ralf W. Grosse-Kunstleve wrote: >>Knowing "I'm spending x% of the time executing Python code" just isn't >>really all that interesting, > > > Right. Sorry if I gave the wrong impression that this could be > interesting. It is indeed not. What is interesting is the estimated > benefit of reimplementing a piece of Python in C/C++. This is in > fact highly correlated with the time/tick. Ah, OK, I think I better understand what you're trying to achieve now. Does knowing that the 2.4 profiler splits out C invocations from the calling Python methods make a difference here? The 2.4 profiler is the only one I've made heavy use of - I'd forgotten that the 2.3 profiler didn't split out the C functions. I've also discovered that the profiler documentation in Python 2.4 still describes the old behaviour (i.e. C-code time getting charged to the calling Python function). (I'll put a bug report on SF for that one) I've got some sample output comparing the 2.3 and 2.4 profilers below. The interesting lines in the 2.4 output are the ones without a filename - those are the C functions. Cheers, Nick. Python 2.3 profiler: ================== $ python /usr/lib/python2.3/profile.py /usr/lib/python2.3/profile.py usage: profile.py scriptfile [arg] ... 6 function calls in 0.031 CPU seconds Ordered by: standard name ncalls tottime percall cumtime percall filename:lineno(function) 1 0.015 0.015 0.015 0.015 <string>:1(?) 1 0.000 0.000 0.000 0.000 profile.py:10(?) 1 0.000 0.000 0.000 0.000 profile.py:103(Profile) 1 0.000 0.000 0.000 0.000 profile.py:334(fake_code) 1 0.000 0.000 0.000 0.000 profile.py:344(fake_frame) 1 0.016 0.016 0.031 0.031 profile:0(execfile('/usr/lib/pytho n2.3/profile.py')) 0 0.000 0.000 profile:0(profiler) ================== And with Python 2.4: ================== $ ./python -m profile -s time /usr/lib/python2.3/profile.py usage: profile.py scriptfile [arg] ... 10 function calls in 0.048 CPU seconds Ordered by: internal time ncalls tottime percall cumtime percall filename:lineno(function) 1 0.032 0.032 0.032 0.032 :0(setprofile) 1 0.016 0.016 0.016 0.016 :0(execfile) 1 0.000 0.000 0.000 0.000 profile.py:334(fake_code) 1 0.000 0.000 0.000 0.000 :0(exit) 1 0.000 0.000 0.000 0.000 profile.py:10(?) 1 0.000 0.000 0.000 0.000 profile.py:344(fake_frame) 1 0.000 0.000 0.000 0.000 :0(hasattr) 1 0.000 0.000 0.000 0.000 profile.py:103(Profile) 1 0.000 0.000 0.048 0.048 profile:0(execfile('/usr/lib/pytho n2.3/profile.py')) 1 0.000 0.000 0.048 0.048 <string>:1(?) 0 0.000 0.000 profile:0(profiler) ================== -- Nick Coghlan | Brisbane, Australia Email: ncoghlan@email.com | Mobile: +61 409 573 268 From s.percivall at chello.se Mon Nov 15 13:31:44 2004 From: s.percivall at chello.se (Simon Percivall) Date: Mon Nov 15 13:31:48 2004 Subject: [Python-Dev] Int literals and method calls In-Reply-To: <20041115095401.GA19806@nl.linux.org> References: <4197A392.7080100@gmx.de> <41982CC2.80006@canterbury.ac.nz> <877e9a1704111420441c9e683c@mail.gmail.com> <419849B3.9020406@v.loewis.de> <20041115095401.GA19806@nl.linux.org> Message-ID: <4F5FA7BF-3702-11D9-B2C1-0003934AD54A@chello.se> On 2004-11-15, at 10.54, Gerrit wrote: > > It works for complex numbers: > >>>> 1j.imag > 1.0 > > ...in a rather surprising way: > >>>> 3+4j.real > 3.0 >>>> f() + 5j.real > 42.0 > > I would have expected it to be 0.0, because I would have expected the > '.' to have precedence over the '+', but it hasn't. apparantly. > > [...] > > I would call it inconsistent: > > 3+4j.real means (3+4j).real > 3+4j.__class__ means 4+(4j).__class__ No, it means 3 + (4j.real) which is the same as 3 + 0.0. Both equals 3.0. //Simon From ncoghlan at gmail.com Mon Nov 15 14:41:59 2004 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon Nov 15 14:42:13 2004 Subject: [Python-Dev] syntactic shortcut - unpack to variably sized list In-Reply-To: <4197B42A.8020401@v.loewis.de> References: <000c01c4c838$77bb3da0$0d00a8c0@MAINFRAME> <4195F883.8030309@iinet.net.au> <20041114131708.GA9442@vicky.ecs.soton.ac.uk> <4197B42A.8020401@v.loewis.de> Message-ID: <4198B227.8080309@iinet.net.au> Martin v. L?wis wrote: > Armin Rigo wrote: > > Of course, whoever implements the change should make sure that slices > are applicable consistently everwhere the documentation says they should > be. Random thought: the two uses of "[" and "]" in the sequence docs can get a little confusing. . . or maybe I'm just tired ;) Anyway, the sequence and mutable sequence sections of the documentation don't reveal anything other than list.pop(). It seems to be the only normal method that accepts an index as an argument. Everything else looks to be dealt with via standard subscripts and the associated magic methods: Retrieval: x = L[i] | x = L.__getitem__(i) Setting: L[i] = x | L.__setitem__(i, x) Deletion: del L[i] | L.__delitem__(i, x) So it looks like list.pop simply got left out because it wasn't a magic method, and the idea of permitting extended slicing for it just didn't come up (well, until now).** Anyway, if this is implemented, array.pop and UserList.pop should allow slices, too, as they are described as working like list.pop. The other container classes I've checked are either immutable, or don't allow indexed access beyond the basics (string, unicode, set, deque). Cheers, Nick. ** Some items of possible historical interest: http://www.python.org/doc/2.3.4/whatsnew/section-slices.html http://sourceforge.net/tracker/?group_id=5470&atid=305470&aid=400998&func=detail http://mail.python.org/pipermail/python-dev/2002-May/023874.html -- Nick Coghlan | Brisbane, Australia Email: ncoghlan@email.com | Mobile: +61 409 573 268 From nbastin at opnet.com Mon Nov 15 16:19:08 2004 From: nbastin at opnet.com (Nick Bastin) Date: Mon Nov 15 16:19:43 2004 Subject: [Python-Dev] proposal+patch: sys.gettickeraccumulation() In-Reply-To: <4198887F.8000100@iinet.net.au> References: <200411141918.iAEJIEvw023160@boa.lbl.gov> <4198887F.8000100@iinet.net.au> Message-ID: <B247747B-3719-11D9-B482-000D932927FE@opnet.com> On Nov 15, 2004, at 5:44 AM, Nick Coghlan wrote: > Ah, OK, I think I better understand what you're trying to achieve now. > Does knowing that the 2.4 profiler splits out C invocations from the > calling Python methods make a difference here? > > The 2.4 profiler is the only one I've made heavy use of - I'd forgotten > that the 2.3 profiler didn't split out the C functions. > > I've also discovered that the profiler documentation in Python 2.4 > still > describes the old behaviour (i.e. C-code time getting charged to the > calling Python function). (I'll put a bug report on SF for that one) Doh, I thought I'd found all of the places in the doc that needed to be corrected. If you file an SF bug, or just let me know where, I'll fix the documentation to reflect the current state of things. -- Nick From fumanchu at amor.org Mon Nov 15 21:02:28 2004 From: fumanchu at amor.org (Robert Brewer) Date: Mon Nov 15 21:04:00 2004 Subject: [Python-Dev] os.access versus os.exist Message-ID: <3A81C87DC164034AA4E2DDFE11D258E3245201@exchange.hqamor.amorhq.net> It would be awfully nice (on posix platforms, for my use-case) to find out whether a file is inaccessible due to permission restrictions, or due to non-existence. Poking around in posixmodule.c, it seems os.access() uses posix_access(PyObject *self, PyObject *args), which calls _access/_waccess as described here: http://msdn.microsoft.com/library/en-us/vclib/html/_crt__access.2c_._wac cess.asp Run-Time Library Reference _access, _waccess Determine file-access permission. int _access( const char *path, int mode ); int _waccess( const wchar_t *path, int mode ); Parameters path File or directory path. mode Permission setting. Return Value Each function returns 0 if the file has the given mode. The function returns -1 if the named file does not exist or is not accessible in the given mode; in this case, errno is set as follows: EACCES Access denied: file's permission setting does not allow specified access. ENOENT Filename or path not found. ----------------- Here's the current posixmodule.posix_access (rev 2.329): static PyObject * posix_access(PyObject *self, PyObject *args) { char *path; int mode; int res; #ifdef Py_WIN_WIDE_FILENAMES if (unicode_file_names()) { PyUnicodeObject *po; if (PyArg_ParseTuple(args, "Ui:access", &po, &mode)) { Py_BEGIN_ALLOW_THREADS /* PyUnicode_AS_UNICODE OK without thread lock as it is a simple dereference. */ res = _waccess(PyUnicode_AS_UNICODE(po), mode); Py_END_ALLOW_THREADS return(PyBool_FromLong(res == 0)); } /* Drop the argument parsing error as narrow strings are also valid. */ PyErr_Clear(); } #endif if (!PyArg_ParseTuple(args, "si:access", &path, &mode)) return NULL; Py_BEGIN_ALLOW_THREADS res = access(path, mode); Py_END_ALLOW_THREADS return(PyBool_FromLong(res == 0)); } ----------------- It seems to me that, if EACCES and ENOENT were made available (in a future version of Python), I could solve my problem easily. One way would be to raise OSError within os.access if the result of _waccess == ENOENT, but this would cause problems with old code which depends upon the old behavior. Can we add an optional "error_if_not_found" arg to os.access? Return the result of _waccess instead of True/False? Make a new os function with the desired behavior? The next issue, then, is other platforms: is similar functionality available in OS/2, unix, etc.? Finally, if anyone has any immediate workarounds for me ;) answer in private or on c.l.p. Robert Brewer MIS Amor Ministries fumanchu@amor.org From rwgk at cci.lbl.gov Mon Nov 15 21:28:19 2004 From: rwgk at cci.lbl.gov (Ralf W. Grosse-Kunstleve) Date: Mon Nov 15 21:28:25 2004 Subject: [Python-Dev] proposal+patch: sys.gettickeraccumulation() Message-ID: <200411152028.iAFKSJgq040962@boa.lbl.gov> Nick Coghlan wrote: > Ralf W. Grosse-Kunstleve wrote: > > >Knowing "I'm spending x% of the time executing Python code" just isn't > > >really all that interesting, > > > > Right. Sorry if I gave the wrong impression that this could be > > interesting. It is indeed not. What is interesting is the estimated > > benefit of reimplementing a piece of Python in C/C++. This is in > > fact highly correlated with the time/tick. > > Ah, OK, I think I better understand what you're trying to achieve now. > Does knowing that the 2.4 profiler splits out C invocations from the > calling Python methods make a difference here? Yes, this is a very helpful improvement. Thanks for pointing this out! It gives us a more direct way of pin-pointing the time sinks. However, I see time.time()/sys.gettickeraccumulation() as a complementary source of information: profiler: factor 8 runtime penalty very detailed information time.time()/sys.gettickeraccumulation(): no runtime penalty global information As we work on new algorithms we can easily monitor the time/tick without a runtime penalty. As long as the ratio stays reasonably high (e.g. because we use numarray-like array operations already coded in C++) we don't have to do anything. If the ratio goes down significantly we can run the profiler to do detailed analysis of the new code. Cheers, Ralf From gvanrossum at gmail.com Mon Nov 15 21:51:44 2004 From: gvanrossum at gmail.com (Guido van Rossum) Date: Mon Nov 15 21:51:47 2004 Subject: [Python-Dev] os.access versus os.exist In-Reply-To: <3A81C87DC164034AA4E2DDFE11D258E3245201@exchange.hqamor.amorhq.net> References: <3A81C87DC164034AA4E2DDFE11D258E3245201@exchange.hqamor.amorhq.net> Message-ID: <ca471dc2041115125131738f0b@mail.gmail.com> > It would be awfully nice (on posix platforms, for my use-case) to find > out whether a file is inaccessible due to permission restrictions, or > due to non-existence. Why can't you solve this by doing a stat() when access() returns False? Also, in general I recommend against using access() except for setuid apps that need to test whether the *real* user id has the desired access; there are many additional reasons why opening a file fails even though access() returns True; amongst the causes can be race conditions (the file was changed after you called access() or conditions like ETXTBSY (you can't write an executable file that is being executed) or ENOMEM (a kernel resource problem). -- --Guido van Rossum (home page: http://www.python.org/~guido/) From martin at v.loewis.de Mon Nov 15 21:58:55 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon Nov 15 21:58:57 2004 Subject: [Python-Dev] Int literals and method calls In-Reply-To: <877e9a170411142330661d3a33@mail.gmail.com> References: <4197A392.7080100@gmx.de> <41982CC2.80006@canterbury.ac.nz> <877e9a1704111420441c9e683c@mail.gmail.com> <419849B3.9020406@v.loewis.de> <877e9a170411142330661d3a33@mail.gmail.com> Message-ID: <4199188F.7000906@v.loewis.de> Michael Walter wrote: > I think it's inconsistent because it works for "list literals" but not > for "integer literals". What do I miss? That tokenization works consistently, using the "maximum match" strategy. If you meant to parse 1.__class__ as "<integer 1>" "." "<identifier __class__>", not as "<float 1.0>" "<identifier __class_>", then, for consistency, you should also parse the second line of s = 100 prints as "<keyword print>" "<identifier s>", not as "<identifier prints>". Since the latter is certainly undesirable, the former must be followed for consistency. You easily derive the rule "a space is necessary between keyword and identifier" from the second example; you should, for consistency, also derive the rule "a space is necessary between an integer literal and a dot". As for "list literals": The Python grammar calls them "displays", not "literals", as they don't (necessarily) denote a literal value, e.g. in [1,2,x,y+5]. Regards, Martin From martin at v.loewis.de Mon Nov 15 22:00:41 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon Nov 15 22:00:42 2004 Subject: [Python-Dev] Int literals and method calls In-Reply-To: <419861AB.6020400@gmx.de> References: <4197A392.7080100@gmx.de> <4197CE89.2060801@v.loewis.de> <419861AB.6020400@gmx.de> Message-ID: <419918F9.3070407@v.loewis.de> Kay Schluehr wrote: > Thank You. I agree completely with Your consistency rule, which is actually > a disambiguation. On the other hand It's not just me who was confused about > method calls on int-literals and for convenience it should be > emphasized, at least > in the language reference but perhaps from the beginning, in the tutorial. Patches to the language reference are welcome. I'm not so sure about patches to the tutorial - this is a fairly infrequent problem, so the tutorial should not bother the reader with this minor nit (IMO). Regards, Martin From martin at v.loewis.de Mon Nov 15 22:34:23 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon Nov 15 22:34:24 2004 Subject: [Python-Dev] syntactic shortcut - unpack to variably sized list In-Reply-To: <4198B227.8080309@iinet.net.au> References: <000c01c4c838$77bb3da0$0d00a8c0@MAINFRAME> <4195F883.8030309@iinet.net.au> <20041114131708.GA9442@vicky.ecs.soton.ac.uk> <4197B42A.8020401@v.loewis.de> <4198B227.8080309@iinet.net.au> Message-ID: <419920DF.9070002@v.loewis.de> Nick Coghlan wrote: > Anyway, the sequence and mutable sequence sections of the documentation > don't reveal anything other than list.pop(). It seems to be the only > normal method that accepts an index as an argument. This isn't really true: s.index(x[, i[, j]]) return smallest k such that s[k] == x and i <= k < j s.insert(i, x) same as s[i:i] = [x] However, I don't think this naturally extends to slices: for index, there is no <= relationship for slices (atleast not a natural one), and for insert, you can use slices as start- and end-index of a slice. > Anyway, if this is implemented, array.pop and UserList.pop should allow > slices, too, as they are described as working like list.pop. Right. For UserList, it probably falls out naturally. Regards, Martin From fumanchu at amor.org Mon Nov 15 22:36:24 2004 From: fumanchu at amor.org (Robert Brewer) Date: Mon Nov 15 22:37:56 2004 Subject: [Python-Dev] os.access versus os.exist Message-ID: <3A81C87DC164034AA4E2DDFE11D258E3245202@exchange.hqamor.amorhq.net> I wrote: > > It would be awfully nice (on posix platforms, for my > > use-case) to find out whether a file is inaccessible > > due to permission restrictions, or due to non-existence. Guido van Rossum replied: > Why can't you solve this by doing a stat() when access() > returns False? In my current case, stat() gives an error (even though the file exists): OSError: [Errno 2] No such file or directory: u'S:/Scans/Projects/2004/TJ/04TJ0267.jpg' Running the same stat() call in the interpreter (as opposed to inside my mod_python app) gives no such error; I get normal stat output, so the file does exist. I figured since the app was running as LocalSystem it was a permissions issue. [One quick sanity check later] Yes, if I run Apache2 under my account, stat() does not error. Hm. I see now I'm following the wrong issue. It has more to do with how Windows shares mapped drives between users (it doesn't). If I use the UNC path, I don't have an issue. Thanks for pointing me away from the wrong direction. ;) Robert Brewer MIS Amor Ministries fumanchu@amor.org From fumanchu at amor.org Mon Nov 15 22:50:10 2004 From: fumanchu at amor.org (Robert Brewer) Date: Mon Nov 15 22:51:42 2004 Subject: [Python-Dev] os.access versus os.exist Message-ID: <3A81C87DC164034AA4E2DDFE11D258E3245203@exchange.hqamor.amorhq.net> Me: > > > It would be awfully nice (on posix platforms, for my > > > use-case) to find out whether a file is inaccessible > > > due to permission restrictions, or due to non-existence. Guido van Rossum: > > Why can't you solve this by doing a stat() when access() > > returns False? Me: > In my current case, stat() gives an error (even though the > file exists): > > OSError: [Errno 2] No such file or directory: > u'S:/Scans/Projects/2004/TJ/04TJ0267.jpg' > > Running the same stat() call in the interpreter (as opposed > to inside my mod_python app) gives no such error; I get normal stat > output, so the file does exist. I figured since the app was running > as LocalSystem it was a permissions issue. [One quick sanity check > later] Yes, if I run Apache2 under my account, stat() does not error. > > Hm. I see now I'm following the wrong issue. It has more to > do with how Windows shares mapped drives between users (it doesn't). > If I use the UNC path, I don't have an issue. Bah. Spoke too soon. I still have the issue even if I use UNC paths. In Pythonwin interactive session (my logon): os.stat() returns a tuple, os.path.exists() returns True, and os.access() returns True. Inside the app (running under LocalSystem on the same Win2k machine): os.stat() raises OSError: [Errno 2] No such file or directory, os.path.exists() returns False, and os.access() returns False. Robert Brewer MIS Amor Ministries fumanchu@amor.org From martin at v.loewis.de Mon Nov 15 22:58:41 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon Nov 15 22:58:42 2004 Subject: [Python-Dev] os.access versus os.exist In-Reply-To: <3A81C87DC164034AA4E2DDFE11D258E3245201@exchange.hqamor.amorhq.net> References: <3A81C87DC164034AA4E2DDFE11D258E3245201@exchange.hqamor.amorhq.net> Message-ID: <41992691.8040708@v.loewis.de> Robert Brewer wrote: > It would be awfully nice (on posix platforms, for my use-case) to find > out whether a file is inaccessible due to permission restrictions, or > due to non-existence. That's easy enough: just check the exception you get when you try opening the file. Don't trust that a negative answer from access() means that you cannot open the file, or that a positive answer means that you can. In short, the implementation of access(2) might lie to you. For example, you might have write access, but writing fails because you are over quota. Or, you are running setuid, and the effective uid is different from the real one. > Can we add an optional "error_if_not_found" arg to > os.access? Make a > new os function with the desired behavior? Either would be fine. Changing the return value would not, since you would have to negate the result. > The next issue, then, is other platforms: is similar functionality > available in OS/2, unix, etc.? On Unix, access(2) can return (in errno) these errors: EACCES, ELOOP, ENAMETOOLONG, ENOENT, ENOTDIR, EROFS, and, optionally, EFAULT, EINVAL, EIO, ENOMEM, ETXTBSY. Regards, Martin From fumanchu at amor.org Mon Nov 15 23:16:13 2004 From: fumanchu at amor.org (Robert Brewer) Date: Mon Nov 15 23:17:45 2004 Subject: [Python-Dev] os.access versus os.exist Message-ID: <3A81C87DC164034AA4E2DDFE11D258E3245204@exchange.hqamor.amorhq.net> Martin v. L?wis wrote: > Robert Brewer wrote: > > It would be awfully nice (on posix platforms, for my > > use-case) to find out whether a file is inaccessible > > due to permission restrictions, or due to non-existence. > > That's easy enough: just check the exception you get when you try > opening the file. Brilliant! I'll do that. I didn't think of open() because I don't need to open it--just see if it's there. /*bonk* self Robert Brewer MIS Amor Ministries fumanchu@amor.org From tdelaney at avaya.com Tue Nov 16 00:04:41 2004 From: tdelaney at avaya.com (Delaney, Timothy C (Timothy)) Date: Tue Nov 16 00:05:12 2004 Subject: [Python-Dev] Int literals and method calls Message-ID: <338366A6D2E2CA4C9DAEAE652E12A1DE4A90AF@au3010avexu1.global.avaya.com> "Martin v. L?wis" wrote: > If you meant to parse > > 1.__class__ > > as "<integer 1>" "." "<identifier __class__>", not as "<float 1.0>" > "<identifier __class_>" On a related note ... All this would just go away if in Python 3.0 (or even earlier ;) floats required something after the decimal point i.e. to get '<float 1.0>' you had to type '1.0' and '1.' by itself was a syntax error. I've never actually seen the use case for having '1.' parse as '<float 1.0>' although I'm sure there is one. Tim Delaney From michael.walter at gmail.com Tue Nov 16 00:07:45 2004 From: michael.walter at gmail.com (Michael Walter) Date: Tue Nov 16 00:07:48 2004 Subject: [Python-Dev] Int literals and method calls In-Reply-To: <4199188F.7000906@v.loewis.de> References: <4197A392.7080100@gmx.de> <41982CC2.80006@canterbury.ac.nz> <877e9a1704111420441c9e683c@mail.gmail.com> <419849B3.9020406@v.loewis.de> <877e9a170411142330661d3a33@mail.gmail.com> <4199188F.7000906@v.loewis.de> Message-ID: <877e9a1704111515073bc2900d@mail.gmail.com> I understand, thanks. - Michael On Mon, 15 Nov 2004 21:58:55 +0100, Martin v. L?wis <martin@v.loewis.de> wrote: > Michael Walter wrote: > > I think it's inconsistent because it works for "list literals" but not > > for "integer literals". What do I miss? > > That tokenization works consistently, using the "maximum match" > strategy. > > If you meant to parse > > 1.__class__ > > as "<integer 1>" "." "<identifier __class__>", not as "<float 1.0>" > "<identifier __class_>", then, for consistency, you should also parse > the second line of > > s = 100 > prints > > as "<keyword print>" "<identifier s>", not as "<identifier prints>". > Since the latter is certainly undesirable, the former must be > followed for consistency. > > You easily derive the rule "a space is necessary between keyword > and identifier" from the second example; you should, for consistency, > also derive the rule "a space is necessary between an integer > literal and a dot". > > As for "list literals": The Python grammar calls them "displays", > not "literals", as they don't (necessarily) denote a literal value, > e.g. in [1,2,x,y+5]. > > Regards, > Martin > From ncoghlan at gmail.com Tue Nov 16 02:34:03 2004 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue Nov 16 02:34:13 2004 Subject: [Python-Dev] syntactic shortcut - unpack to variably sized list In-Reply-To: <419920DF.9070002@v.loewis.de> References: <000c01c4c838$77bb3da0$0d00a8c0@MAINFRAME> <4195F883.8030309@iinet.net.au> <20041114131708.GA9442@vicky.ecs.soton.ac.uk> <4197B42A.8020401@v.loewis.de> <4198B227.8080309@iinet.net.au> <419920DF.9070002@v.loewis.de> Message-ID: <4199590B.1040908@iinet.net.au> Martin v. L?wis wrote: > Nick Coghlan wrote: > >> Anyway, the sequence and mutable sequence sections of the >> documentation don't reveal anything other than list.pop(). It seems to >> be the only normal method that accepts an index as an argument. > > > This isn't really true: > > s.index(x[, i[, j]]) > return smallest k such that s[k] == x and i <= k < j > s.insert(i, x) > same as s[i:i] = [x] > > However, I don't think this naturally extends to slices: > for index, there is no <= relationship for slices (atleast > not a natural one), and for insert, you can use slices as > start- and end-index of a slice. Yes. Those two were on my list initially, but then I tried to figure out how using a slice would actually *work* for them. At which point, I took them back off the list - slice arguments just didn't make any sense. So I think we're down to two things to implement - list.pop and array.pop. As you say, UserList.pop should just be a different way of spelling list.pop. Cheers, Nick. -- Nick Coghlan | Brisbane, Australia Email: ncoghlan@email.com | Mobile: +61 409 573 268 From greg.ewing at canterbury.ac.nz Tue Nov 16 02:50:53 2004 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue Nov 16 02:51:00 2004 Subject: [Python-Dev] os.access versus os.exist In-Reply-To: <3A81C87DC164034AA4E2DDFE11D258E3245201@exchange.hqamor.amorhq.net> References: <3A81C87DC164034AA4E2DDFE11D258E3245201@exchange.hqamor.amorhq.net> Message-ID: <41995CFD.6030902@canterbury.ac.nz> Robert Brewer wrote: > It would be awfully nice (on posix platforms, for my use-case) to find > out whether a file is inaccessible due to permission restrictions, or > due to non-existence. On a posix system, you should be able to find out by checking the error code from the exception. That would be better than making a separate call, since it would avoid any possible race conditions. Don't look before you leap, etc... -- Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From greg.ewing at canterbury.ac.nz Tue Nov 16 03:00:53 2004 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue Nov 16 03:01:00 2004 Subject: [Python-Dev] Int literals and method calls In-Reply-To: <338366A6D2E2CA4C9DAEAE652E12A1DE4A90AF@au3010avexu1.global.avaya.com> References: <338366A6D2E2CA4C9DAEAE652E12A1DE4A90AF@au3010avexu1.global.avaya.com> Message-ID: <41995F55.8040705@canterbury.ac.nz> Delaney, Timothy C (Timothy) wrote: > All this would just go away if in Python 3.0 (or even earlier ;) > floats required something after the decimal point i.e. to get > '<float 1.0>' you had to type '1.0' and '1.' by itself was a > syntax error. I always write float literals that way anyhow, in the interests of clarity. So I wouldn't mind if this happened either. +0 -- Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From ncoghlan at gmail.com Tue Nov 16 03:02:39 2004 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue Nov 16 03:02:45 2004 Subject: [Python-Dev] proposal+patch: sys.gettickeraccumulation() In-Reply-To: <200411152028.iAFKSJgq040962@boa.lbl.gov> References: <200411152028.iAFKSJgq040962@boa.lbl.gov> Message-ID: <41995FBF.3080207@iinet.net.au> Ralf W. Grosse-Kunstleve wrote: > As we work on new algorithms we can easily monitor the time/tick > without a runtime penalty. As long as the ratio stays reasonably high > (e.g. because we use numarray-like array operations already coded in > C++) we don't have to do anything. If the ratio goes down significantly > we can run the profiler to do detailed analysis of the new code. That seems like a reasonable approach to me - one could easily make use of the ticker information in a test framework. So I'll withdraw my objection and give the new function a +0. Cheers, Nick. -- Nick Coghlan | Brisbane, Australia Email: ncoghlan@email.com | Mobile: +61 409 573 268 From ncoghlan at gmail.com Tue Nov 16 03:22:55 2004 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue Nov 16 03:23:01 2004 Subject: [Python-Dev] proposal+patch: sys.gettickeraccumulation() In-Reply-To: <B247747B-3719-11D9-B482-000D932927FE@opnet.com> References: <200411141918.iAEJIEvw023160@boa.lbl.gov> <4198887F.8000100@iinet.net.au> <B247747B-3719-11D9-B482-000D932927FE@opnet.com> Message-ID: <4199647F.7000309@iinet.net.au> Nick Bastin wrote: > > On Nov 15, 2004, at 5:44 AM, Nick Coghlan wrote: >> I've also discovered that the profiler documentation in Python 2.4 still >> describes the old behaviour (i.e. C-code time getting charged to the >> calling Python function). (I'll put a bug report on SF for that one) > > > Doh, I thought I'd found all of the places in the doc that needed to be > corrected. If you file an SF bug, or just let me know where, I'll fix > the documentation to reflect the current state of things. The bug report is on SF as of last night. . . *checks browser history for the number*. . . it's bug #1066607. Section 10.6 is the one on Limitations. Also, the front page of the hotshot documentation could probably use a warning that the actual profile module copes with C extensions better than hotshot does (I think hotshot still has the old "C time goes to the calling Python function" behaviour - I haven't checked that recently, though) Cheers, Nick. -- Nick Coghlan | Brisbane, Australia Email: ncoghlan@email.com | Mobile: +61 409 573 268 From bob at redivi.com Tue Nov 16 05:16:21 2004 From: bob at redivi.com (Bob Ippolito) Date: Tue Nov 16 05:17:06 2004 Subject: [Python-Dev] Int literals and method calls In-Reply-To: <41995F55.8040705@canterbury.ac.nz> References: <338366A6D2E2CA4C9DAEAE652E12A1DE4A90AF@au3010avexu1.global.avaya.com> <41995F55.8040705@canterbury.ac.nz> Message-ID: <457ECE6E-3786-11D9-9BE2-000A9567635C@redivi.com> On Nov 16, 2004, at 4:00 AM, Greg Ewing wrote: > Delaney, Timothy C (Timothy) wrote: >> All this would just go away if in Python 3.0 (or even earlier ;) > > floats required something after the decimal point i.e. to get >> '<float 1.0>' you had to type '1.0' and '1.' by itself was a > > syntax error. > > I always write float literals that way anyhow, in the > interests of clarity. So I wouldn't mind if this > happened either. > > +0 I tend to use integer float literals without the trailing zero from the interpreter if I want float division to happen without typing "from __future__ import division". Having to type an additional character isn't a problem, though. +0 -bob From carribeiro at gmail.com Tue Nov 16 13:13:29 2004 From: carribeiro at gmail.com (Carlos Ribeiro) Date: Tue Nov 16 13:13:32 2004 Subject: [Python-Dev] Looking for authoritative documentation on packages, import & ihooks In-Reply-To: <864d3709041115075629d15045@mail.gmail.com> References: <864d3709041115075629d15045@mail.gmail.com> Message-ID: <864d37090411160413756ff4c5@mail.gmail.com> I have posted this question on c.l.py yesterday but received no answers. Given the complexity of the theme, I hope that people will not mind if I ask it here. I am looking for information on packages & import hooks, including simple examples on how to implement a simple import hook. Quick googling turns out some documents, such as: -- PEP 302: http://www.python.org/peps/pep-0302.html -- What's new on Python 2.3: http://www.python.org/doc/2.3.4/whatsnew/section-pep302.html -- Import SIG: http://www.python.org/sigs/import-sig/ -- PyDoc for the ihooks module: http://pydoc.org/2.3/ihooks.html However, after reading them, I ended up more confused than before. There are lots of references to the ihooks standard module, but there is no documentation about it on the standard Python docs. The "Modules That Needs Docs" Wiki entry (http://www.python.org/moin/ModulesThatNeedDocs) lists ihooks and imputils, but noone has volunteered to do it yet (worse, not even a bug report was filled; this is something that I'm going to do). Packages are also not documented (there is still a reference to an essay from 1.5 days on the Python 2.3 docs). For what I could see, the import system went over several iterations and patches since the 1.5 release (when packages were added). It's now hard to understand what is the preferred hooking mechanism as far as Python 2.4 (and future versions) are concerned. For example: it is not clear, from a simple reading, if PEP 302 is up-to-date, or if it is considered the preferred approach for import hooks. For this reason, I would like to know if there is any document which may be considered "authoritative" for the import system (besides ihooks.py and import.c). If *all* that exists is the source code, well, I guess I'll have to read it. But anyway... pointers to simple examples also are helpful. Thanks in advance. -- Carlos Ribeiro Consultoria em Projetos blog: http://rascunhosrotos.blogspot.com blog: http://pythonnotes.blogspot.com mail: carribeiro@gmail.com mail: carribeiro@yahoo.com From pje at telecommunity.com Tue Nov 16 16:50:51 2004 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue Nov 16 16:49:48 2004 Subject: [Python-Dev] Looking for authoritative documentation on packages, import & ihooks In-Reply-To: <864d37090411160413756ff4c5@mail.gmail.com> References: <864d3709041115075629d15045@mail.gmail.com> <864d3709041115075629d15045@mail.gmail.com> Message-ID: <5.1.1.6.0.20041116104453.02dbe740@mail.telecommunity.com> At 10:13 AM 11/16/04 -0200, Carlos Ribeiro wrote: >For example: it is not >clear, from a simple reading, if PEP 302 is up-to-date, or if it is >considered the preferred approach for import hooks. For this reason, I >would like to know if there is any document which may be considered >"authoritative" for the import system (besides ihooks.py and >import.c). If *all* that exists is the source code, well, I guess I'll >have to read it. But anyway... pointers to simple examples also are >helpful. Thanks in advance. PEP 302 is essentially up-to-date (although not fully implemented; see the notes in the PEP), and yes, it's the preferred approach, at least if you're 1) developing an import hook, 2) you're focused on "location" extensions, i.e. importing from places not currently supported, as opposed to adding new filetypes or new ways of compiling, etc., and 3) want to interoperate well with other import hooks. Regarding the use of other import hook modules such as ihooks and imputil, they have in the past been the subject of vigorous debate as to which is the "preferred" way of implementing import hooks. There is also a separately distributed import hook module, 'iu', that inspired the current PEP 302 approach. With respect to these various modules, it is apparently a matter of programmer taste and/or goals as to which is the best. From michael.walter at gmail.com Tue Nov 16 17:01:30 2004 From: michael.walter at gmail.com (Michael Walter) Date: Tue Nov 16 17:01:34 2004 Subject: [Python-Dev] PEP 310 Status Message-ID: <877e9a1704111608015caff2fa@mail.gmail.com> Hello everyone, I was just wondering about the status of PEP 310 ("with" statement) - has there been any concensus/plan to implement it? (I tried to google the answer, but failed ;-) How about the potentiel inclusion of user-defined "blocks"? I suppose this would be only a Python 3000 thing, if ever included? Cheers, Michael From pje at telecommunity.com Tue Nov 16 17:27:58 2004 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue Nov 16 17:26:55 2004 Subject: [Python-Dev] PEP 310 Status In-Reply-To: <877e9a1704111608015caff2fa@mail.gmail.com> Message-ID: <5.1.1.6.0.20041116112416.02705890@mail.telecommunity.com> At 11:01 AM 11/16/04 -0500, Michael Walter wrote: >Hello everyone, > >I was just wondering about the status of PEP 310 ("with" statement) - >has there been any concensus/plan to implement it? (I tried to google >the answer, but failed ;-) Guido has previously said that "with" is reserved for a use similar to the use of the "with" statement in Turbo Pascal and Visual Basic; that is, a shortcut for referring to a long expression. Whether that means PEP 310 needs a new keyword, or that it needs to play a dual role, I'm not sure. >How about the potentiel inclusion of user-defined "blocks"? I suppose >this would be only a Python 3000 thing, if ever included? With the advent of PEP 318, it's now possible to do block-like things with closures and decorators, e.g.: @with_lock(mylock) def do_something(): # code here do_something() This is somewhat ugly, however, when used in-line. On the other hand, this could also be viewed as an encouragement to modularize more, in order to have more readable code. :) From michael.walter at gmail.com Tue Nov 16 17:35:57 2004 From: michael.walter at gmail.com (Michael Walter) Date: Tue Nov 16 17:35:58 2004 Subject: [Python-Dev] PEP 310 Status In-Reply-To: <5.1.1.6.0.20041116112416.02705890@mail.telecommunity.com> References: <877e9a1704111608015caff2fa@mail.gmail.com> <5.1.1.6.0.20041116112416.02705890@mail.telecommunity.com> Message-ID: <877e9a1704111608352f87aa94@mail.gmail.com> This doesn't quite work for different use cases of blocks, such as generating markup using something like "with <tag>: <suite>". Of course one (could|should|will) argue that this is an attempt to extend Python's syntax, basically "abusing" with beyond its original scope. I don't know whether this is a bad thing. Cheers, Michael On Tue, 16 Nov 2004 11:27:58 -0500, Phillip J. Eby <pje@telecommunity.com> wrote: > At 11:01 AM 11/16/04 -0500, Michael Walter wrote: > >Hello everyone, > > > >I was just wondering about the status of PEP 310 ("with" statement) - > >has there been any concensus/plan to implement it? (I tried to google > >the answer, but failed ;-) > > Guido has previously said that "with" is reserved for a use similar to the > use of the "with" statement in Turbo Pascal and Visual Basic; that is, a > shortcut for referring to a long expression. Whether that means PEP 310 > needs a new keyword, or that it needs to play a dual role, I'm not sure. > > > >How about the potentiel inclusion of user-defined "blocks"? I suppose > >this would be only a Python 3000 thing, if ever included? > > With the advent of PEP 318, it's now possible to do block-like things with > closures and decorators, e.g.: > > @with_lock(mylock) > def do_something(): > # code here > > do_something() > > This is somewhat ugly, however, when used in-line. On the other hand, this > could also be viewed as an encouragement to modularize more, in order to > have more readable code. :) > > From pedronis at bluewin.ch Tue Nov 16 17:42:57 2004 From: pedronis at bluewin.ch (Samuele Pedroni) Date: Tue Nov 16 17:39:31 2004 Subject: [Python-Dev] PEP 310 Status In-Reply-To: <877e9a1704111608015caff2fa@mail.gmail.com> References: <877e9a1704111608015caff2fa@mail.gmail.com> Message-ID: <419A2E11.8020100@bluewin.ch> Michael Walter wrote: > Hello everyone, > > I was just wondering about the status of PEP 310 ("with" statement) - > has there been any concensus/plan to implement it? (I tried to google > the answer, but failed ;-) I suppose someone should write an implementation for it, and start championing for it again (And a new keyword may be needed because 'with' is preempted). The lingering suspect that a more general solution could enable it as application and more has worked a bit against it, also by > How about the potentiel inclusion of user-defined "blocks"? I suppose > this would be only a Python 3000 thing, if ever included? > generating enormous discussions about this one other topic that went mostly nowhere. Don't know whether having another one is a good idea. Beforehand someone should write a PEP with a concrete proposal in the fullness of its details, or maybe even a set of different PEP/proposals. There is no self-evident and beautiful solution for this, it seems. Because of some aspects of Python ( focus on readability, non uniformity (think vs Lisp/Smalltalk) e.g. expression vs statement distinction, class def suite semantics vs. functions definitions..., generators design decisions) this will very likely entail some sort of compromise. Depending of how much expressiveness one wants, because of non-uniformity, the "right" solution may be even be macros, not just blocks. For blocks things like: - are we up to blurring a bit the expression vs. statement divide, - can we live with a bit lisp-looking parenthesized constructs in expressions, - Or limiting things to just _one_ block "capture" statement is enough - do we want to support class-def-like constructs? interested in capturing order? markup construction? - do we want anonymous function vs. suite semantics? - what the is interaction with yield? - or with ideas and likely need of rebinding for lexical-scoped variables - etc should be considered/addressed regards From sxanth at ceid.upatras.gr Tue Nov 16 19:51:06 2004 From: sxanth at ceid.upatras.gr (Stelios Xanthakis) Date: Tue Nov 16 18:56:30 2004 Subject: [Python-Dev] __pycode__ extension Message-ID: <419A4C1A.5060502@ceid.upatras.gr> Hi. I posted a message on c.l.p a couple of days ago about a python patch which adds a member __pycde__ to functions and classes. This member is a string which holds the python code of the function/class. (It works in interactively defined code and exec'd definitions) Supposing that: I ported this to 2.4b2 and made it have zero overhead when python runs in normal mode, is there any chance it would be considered a candidate for inclusion in mainline python? If yes, I'd appreciate it if somebody took a look at the patch because there may be a better way to do it. Patch vs 2.3.4: http://students.ceid.upatras.gr/~sxanth/ISYSTEM/python-PIESS.tar.bz2 Guido? c-ya Stelios From aahz at pythoncraft.com Tue Nov 16 19:45:48 2004 From: aahz at pythoncraft.com (Aahz) Date: Tue Nov 16 19:45:51 2004 Subject: [Python-Dev] __pycode__ extension In-Reply-To: <419A4C1A.5060502@ceid.upatras.gr> References: <419A4C1A.5060502@ceid.upatras.gr> Message-ID: <20041116184548.GA5013@panix.com> On Tue, Nov 16, 2004, Stelios Xanthakis wrote: > > I posted a message on c.l.p a couple of days ago about a > python patch which adds a member __pycde__ to functions and > classes. This member is a string which holds the python code of > the function/class. (It works in interactively defined code > and exec'd definitions) > > Supposing that: I ported this to 2.4b2 and made it have zero overhead > when python runs in normal mode, is there any chance it would be > considered a candidate for inclusion in mainline python? There is zero chance it'll go into 2.4. There's a decent chance it'll go into 2.5, but if you get any pushback on the feature, be prepared to write up a full PEP. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ WiFi is the SCSI of the 21st Century -- there are fundamental technical reasons for sacrificing a goat. (with no apologies to John Woods) From jhylton at gmail.com Tue Nov 16 20:01:47 2004 From: jhylton at gmail.com (Jeremy Hylton) Date: Tue Nov 16 20:03:24 2004 Subject: [Python-Dev] __pycode__ extension In-Reply-To: <419A4C1A.5060502@ceid.upatras.gr> References: <419A4C1A.5060502@ceid.upatras.gr> Message-ID: <e8bf7a5304111611013ad74af5@mail.gmail.com> On Tue, 16 Nov 2004 10:51:06 -0800, Stelios Xanthakis <sxanth@ceid.upatras.gr> wrote: > > Hi. > > I posted a message on c.l.p a couple of days ago about a > python patch which adds a member __pycde__ to functions and > classes. This member is a string which holds the python code of > the function/class. (It works in interactively defined code > and exec'd definitions) Functions already have a reference to code objects. How is this different? Jeremy From martin at v.loewis.de Tue Nov 16 20:16:40 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue Nov 16 20:16:42 2004 Subject: [Python-Dev] os.access versus os.exist In-Reply-To: <3A81C87DC164034AA4E2DDFE11D258E3245203@exchange.hqamor.amorhq.net> References: <3A81C87DC164034AA4E2DDFE11D258E3245203@exchange.hqamor.amorhq.net> Message-ID: <419A5218.9090500@v.loewis.de> Robert Brewer wrote: > Inside the app (running under LocalSystem on the same Win2k machine): > os.stat() raises OSError: [Errno 2] No such file or directory, > os.path.exists() returns False, and > os.access() returns False. LocalSystem cannot access network shares. However, this is OT for python-dev. Regards, Martin From martin at v.loewis.de Tue Nov 16 20:20:20 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue Nov 16 20:20:26 2004 Subject: [Python-Dev] Looking for authoritative documentation on packages, import & ihooks In-Reply-To: <864d37090411160413756ff4c5@mail.gmail.com> References: <864d3709041115075629d15045@mail.gmail.com> <864d37090411160413756ff4c5@mail.gmail.com> Message-ID: <419A52F4.2020408@v.loewis.de> Carlos Ribeiro wrote: > For this reason, I > would like to know if there is any document which may be considered > "authoritative" for the import system (besides ihooks.py and > import.c). This question is OT for python-dev. Python is open source, so what is wrong with reading import.c? Regards, Martin From pje at telecommunity.com Tue Nov 16 20:21:46 2004 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue Nov 16 20:20:43 2004 Subject: [Python-Dev] __pycode__ extension In-Reply-To: <e8bf7a5304111611013ad74af5@mail.gmail.com> References: <419A4C1A.5060502@ceid.upatras.gr> <419A4C1A.5060502@ceid.upatras.gr> Message-ID: <5.1.1.6.0.20041116142008.03230a40@mail.telecommunity.com> At 02:01 PM 11/16/04 -0500, Jeremy Hylton wrote: >On Tue, 16 Nov 2004 10:51:06 -0800, Stelios Xanthakis ><sxanth@ceid.upatras.gr> wrote: > > > > Hi. > > > > I posted a message on c.l.p a couple of days ago about a > > python patch which adds a member __pycde__ to functions and > > classes. This member is a string which holds the python code of > > the function/class. (It works in interactively defined code > > and exec'd definitions) > >Functions already have a reference to code objects. How is this different? I believe he means the Python source code; I personally would suggest a name like '__source__' instead of '__pycode__' as the attribute name. Of course, I'd also almost prefer to have the AST than the original source... ;) From sxanth at ceid.upatras.gr Tue Nov 16 20:21:41 2004 From: sxanth at ceid.upatras.gr (Stelios Xanthakis) Date: Tue Nov 16 20:21:49 2004 Subject: [Python-Dev] __pycode__ extension In-Reply-To: <e8bf7a5304111611013ad74af5@mail.gmail.com> References: <419A4C1A.5060502@ceid.upatras.gr> <e8bf7a5304111611013ad74af5@mail.gmail.com> Message-ID: <Pine.GSO.4.61.0411162114260.1618@zenon.ceid.upatras.gr> On Tue, 16 Nov 2004, Jeremy Hylton wrote: > Functions already have a reference to code objects. How is this different? This is the python-code text. >>> def foo (): ... # comment -- instead of doc ... print 'santa' ... >>> print foo.__pycode__ def foo(): # comment -- instead of doc print 'santa' This is fun as I explained in the post at c.l.p because with a function: def save(FILE): for a in globals(): if type(eval(a)) == FunctionType: FILE.write (eval(a).__pycode__) We can save the entire state in a file and continue from there next time. Stelios From sxanth at ceid.upatras.gr Tue Nov 16 20:26:19 2004 From: sxanth at ceid.upatras.gr (Stelios Xanthakis) Date: Tue Nov 16 20:26:25 2004 Subject: [Python-Dev] __pycode__ extension In-Reply-To: <5.1.1.6.0.20041116142008.03230a40@mail.telecommunity.com> References: <419A4C1A.5060502@ceid.upatras.gr> <419A4C1A.5060502@ceid.upatras.gr> <5.1.1.6.0.20041116142008.03230a40@mail.telecommunity.com> Message-ID: <Pine.GSO.4.61.0411162124350.1642@zenon.ceid.upatras.gr> On Tue, 16 Nov 2004, Phillip J. Eby wrote: > > I believe he means the Python source code; I personally would suggest a name > like '__source__' instead of '__pycode__' as the attribute name. > Yes. I'm terrible at choosing names for things :) > Of course, I'd also almost prefer to have the AST than the original source... If you have the python source you can get AST. AST OTOH discards comments and other information. Stelios From jhylton at gmail.com Tue Nov 16 20:35:46 2004 From: jhylton at gmail.com (Jeremy Hylton) Date: Tue Nov 16 20:35:51 2004 Subject: [Python-Dev] Looking for authoritative documentation on packages, import & ihooks In-Reply-To: <419A52F4.2020408@v.loewis.de> References: <864d3709041115075629d15045@mail.gmail.com> <864d37090411160413756ff4c5@mail.gmail.com> <419A52F4.2020408@v.loewis.de> Message-ID: <e8bf7a5304111611354643e892@mail.gmail.com> On Tue, 16 Nov 2004 20:20:20 +0100, Martin v. L?wis <martin@v.loewis.de> wrote: > Carlos Ribeiro wrote: > > For this reason, I > > would like to know if there is any document which may be considered > > "authoritative" for the import system (besides ihooks.py and > > import.c). > > This question is OT for python-dev. > > Python is open source, so what is wrong with reading import.c? import.c isn't a document that purports to describe the interface that import will support over time. import.c has surely got good clues, but I don't find it particularly easy to read. It's a snapshot of code that may or may not implement that interface correctly. The question seems OT to me. Jeremy From pedronis at bluewin.ch Tue Nov 16 20:40:35 2004 From: pedronis at bluewin.ch (Samuele Pedroni) Date: Tue Nov 16 20:37:30 2004 Subject: [Python-Dev] __pycode__ extension In-Reply-To: <5.1.1.6.0.20041116142008.03230a40@mail.telecommunity.com> References: <419A4C1A.5060502@ceid.upatras.gr> <419A4C1A.5060502@ceid.upatras.gr> <5.1.1.6.0.20041116142008.03230a40@mail.telecommunity.com> Message-ID: <419A57B3.3070600@bluewin.ch> Phillip J. Eby wrote: > At 02:01 PM 11/16/04 -0500, Jeremy Hylton wrote: > >> On Tue, 16 Nov 2004 10:51:06 -0800, Stelios Xanthakis >> <sxanth@ceid.upatras.gr> wrote: >> > >> > Hi. >> > >> > I posted a message on c.l.p a couple of days ago about a >> > python patch which adds a member __pycde__ to functions and >> > classes. This member is a string which holds the python code of >> > the function/class. (It works in interactively defined code >> > and exec'd definitions) >> >> Functions already have a reference to code objects. How is this >> different? > > > I believe he means the Python source code; I personally would suggest a > name like '__source__' instead of '__pycode__' as the attribute name. > > Of course, I'd also almost prefer to have the AST than the original > source... ;) a prerequiste for that is likely the ast-branch, because the old "A"STs are not a reasonable format to be mandated for all python implementations. From fperez528 at yahoo.com Tue Nov 16 20:38:52 2004 From: fperez528 at yahoo.com (Fernando Perez) Date: Tue Nov 16 20:39:06 2004 Subject: [Python-Dev] Re: __pycode__ extension References: <419A4C1A.5060502@ceid.upatras.gr> <20041116184548.GA5013@panix.com> Message-ID: <cndl0c$cr8$1@sea.gmane.org> Aahz wrote: > On Tue, Nov 16, 2004, Stelios Xanthakis wrote: >> >> I posted a message on c.l.p a couple of days ago about a >> python patch which adds a member __pycde__ to functions and >> classes. This member is a string which holds the python code of >> the function/class. (It works in interactively defined code >> and exec'd definitions) >> >> Supposing that: I ported this to 2.4b2 and made it have zero overhead >> when python runs in normal mode, is there any chance it would be >> considered a candidate for inclusion in mainline python? > > There is zero chance it'll go into 2.4. There's a decent chance it'll > go into 2.5, but if you get any pushback on the feature, be prepared to > write up a full PEP. Well, not necessarily pushback, but I'd like a clarification at least. What kind of memory overhead does this introduce? If every function running around is holding a full copy of all its source, is this overhead potentially significant? What happens with decorators, which modify functions but are not explicit source-level transformations? Since this is already fairly straightforward to implement via inspect, I'd like to see a pretty strong justification for its real need before seeing it go in. It smells a bit of unnecessary bloat to me, but perhaps I'm missing something obvious, so I'd be happy to be educated. Regards, f From tim.peters at gmail.com Tue Nov 16 20:45:24 2004 From: tim.peters at gmail.com (Tim Peters) Date: Tue Nov 16 20:45:27 2004 Subject: [Python-Dev] Looking for authoritative documentation on packages, import & ihooks In-Reply-To: <419A52F4.2020408@v.loewis.de> References: <864d3709041115075629d15045@mail.gmail.com> <864d37090411160413756ff4c5@mail.gmail.com> <419A52F4.2020408@v.loewis.de> Message-ID: <1f7befae0411161145182e417c@mail.gmail.com> [Carlos Ribeiro] ... >> For this reason, I would like to know if there is any document which may >> be considered "authoritative" for the import system (besides ihooks.py >> and import.c). [Martin v. L?wis] > This question is OT for python-dev. > > Python is open source, so what is wrong with reading import.c? I've tried. I usually stop when the pool of blood from my eyeballs rises above my ankles <0.9 wink>. I'm not sure this is OT for python-dev. There are many cases involving circular and kinda-circular imports, with and without packages, where the Python docs say nothing about what must, should, or can happen. So we still get bug reports like this recent one, where nobody has a definitive answer: http://www.python.org/sf/992389 That case doesn't involve import hooks, just the meaning of package imports and what you can and can't "get away with" in __init__.py files. In the presence of user-defined import hooks too, reaching the clarity of mud in this case would be a noteworthy accomplishment. From martin at v.loewis.de Tue Nov 16 20:58:45 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue Nov 16 20:58:48 2004 Subject: [Python-Dev] Looking for authoritative documentation on packages, import & ihooks In-Reply-To: <1f7befae0411161145182e417c@mail.gmail.com> References: <864d3709041115075629d15045@mail.gmail.com> <864d37090411160413756ff4c5@mail.gmail.com> <419A52F4.2020408@v.loewis.de> <1f7befae0411161145182e417c@mail.gmail.com> Message-ID: <419A5BF5.4090201@v.loewis.de> Tim Peters wrote: > I'm not sure this is OT for python-dev. There are many cases > involving circular and kinda-circular imports, with and without > packages, where the Python docs say nothing about what must, should, > or can happen. Not sure what "this" is, here. The import machinery certainly is on-topic. The question "where do I find more documentation" is off-topic. The question "I have this and that change to the documentation, do people agree it is correct" would be on-topic. Regards, Martin From tim.peters at gmail.com Tue Nov 16 21:05:01 2004 From: tim.peters at gmail.com (Tim Peters) Date: Tue Nov 16 21:05:03 2004 Subject: [Python-Dev] Looking for authoritative documentation on packages, import & ihooks In-Reply-To: <419A5BF5.4090201@v.loewis.de> References: <864d3709041115075629d15045@mail.gmail.com> <864d37090411160413756ff4c5@mail.gmail.com> <419A52F4.2020408@v.loewis.de> <1f7befae0411161145182e417c@mail.gmail.com> <419A5BF5.4090201@v.loewis.de> Message-ID: <1f7befae04111612057efb142@mail.gmail.com> [Tim Peters] >> I'm not sure this is OT for python-dev. There are many cases >> involving circular and kinda-circular imports, with and without >> packages, where the Python docs say nothing about what must, should, >> or can happen. [Martin v. L?wis] > Not sure what "this" is, here. Ya, and I'm not sure what "OT" meant anywhere -- "on" and "off" both start with "O" in English <wink>. > The import machinery certainly is on-topic. The question "where do I find more > documentation" is off-topic. Could be, but the "sorry, there isn't more" answer seems on-topic anyway: developers need to be reminded about areas that remain ill-defined. > The question "I have this and that change to the documentation, do people > agree it is correct" would be on-topic. Yup, that too. From pje at telecommunity.com Tue Nov 16 21:19:34 2004 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue Nov 16 21:18:32 2004 Subject: [Python-Dev] Looking for authoritative documentation on packages, import & ihooks In-Reply-To: <419A5BF5.4090201@v.loewis.de> References: <1f7befae0411161145182e417c@mail.gmail.com> <864d3709041115075629d15045@mail.gmail.com> <864d37090411160413756ff4c5@mail.gmail.com> <419A52F4.2020408@v.loewis.de> <1f7befae0411161145182e417c@mail.gmail.com> Message-ID: <5.1.1.6.0.20041116150836.02b32cc0@mail.telecommunity.com> At 08:58 PM 11/16/04 +0100, Martin v. L?wis wrote: >Tim Peters wrote: >>I'm not sure this is OT for python-dev. There are many cases >>involving circular and kinda-circular imports, with and without >>packages, where the Python docs say nothing about what must, should, >>or can happen. > >Not sure what "this" is, here. The import machinery certainly is >on-topic. The question "where do I find more documentation" is >off-topic. That was not what the OP asked; the clear intent of the request was what documentation should be considered *authoritative* on the matter, as well as what approaches were to be *preferred*. IMO those sound like items that request the judgment and/or "blessing" of the Python development team, in the absence of any clearly authoritative documents or statements like "such-and-such is the new approach, so-and-so is the old". Further, the OP: 1. Requested info on python-list first 2. Googled for, and read, the appropriate documentation 3. Raised issues regarding the documentation, and stated an intent to file bug reports with more specifics ISTM these good faith efforts should give someone a little bit more benefit of the doubt in a borderline case... not that this particular case seemed at all borderline to me. From fumanchu at amor.org Tue Nov 16 22:59:24 2004 From: fumanchu at amor.org (Robert Brewer) Date: Tue Nov 16 23:00:58 2004 Subject: [Python-Dev] os.access versus os.exist Message-ID: <3A81C87DC164034AA4E2DDFE11D258E3245214@exchange.hqamor.amorhq.net> Martin v. L?wis wrote: > Robert Brewer wrote: > > Inside the app (running under LocalSystem on the same Win2k > machine): > > os.stat() raises OSError: [Errno 2] No such file or directory, > > os.path.exists() returns False, and > > os.access() returns False. > > LocalSystem cannot access network shares. However, this is OT for > python-dev. Completely understood about LocalSystem. But IMO, os.stat should raise a more specific and correct error. As you pointed out, calling open() raises "IOError: [Errno 13] Permission denied"; it would be nice if stat did the same (in my example). The platform-specific API call (for example, Windows _stati64) doesn't provide this--it would require another test inside posix_do_stat to return a more appropriate error. But it would be a nice touch for us end-users. In my opinion, "no such file" is the wrong error message and is therefore a bug. But I can accept others disagreeing, since os.stat() simply wraps Windows' _stat* calls (at least, in my example). Robert Brewer MIS Amor Ministries fumanchu@amor.org From mwh at python.net Tue Nov 16 23:46:02 2004 From: mwh at python.net (Michael Hudson) Date: Tue Nov 16 23:46:06 2004 Subject: [Python-Dev] Looking for authoritative documentation on packages, import & ihooks In-Reply-To: <1f7befae0411161145182e417c@mail.gmail.com> (Tim Peters's message of "Tue, 16 Nov 2004 14:45:24 -0500") References: <864d3709041115075629d15045@mail.gmail.com> <864d37090411160413756ff4c5@mail.gmail.com> <419A52F4.2020408@v.loewis.de> <1f7befae0411161145182e417c@mail.gmail.com> Message-ID: <2mu0rpza39.fsf@starship.python.net> Tim Peters <tim.peters@gmail.com> writes: > [Carlos Ribeiro] > ... >>> For this reason, I would like to know if there is any document which may >>> be considered "authoritative" for the import system (besides ihooks.py >>> and import.c). > > [Martin v. L?wis] >> This question is OT for python-dev. >> >> Python is open source, so what is wrong with reading import.c? > > I've tried. I usually stop when the pool of blood from my eyeballs > rises above my ankles <0.9 wink>. I'm glad to hear this isn't just me :-) Cheers, mwh (officially scared of Python/import.c for some time now) -- <dash> if python is an orchestra, overloaded operators are "miscellaneous percussion" -- from Twisted.Quotes From kbk at shore.net Wed Nov 17 06:28:08 2004 From: kbk at shore.net (Kurt B. Kaiser) Date: Wed Nov 17 06:28:14 2004 Subject: [Python-Dev] Weekly Python Patch/Bug Summary Message-ID: <200411170528.iAH5S8E3026597@h006008a7bda6.ne.client2.attbi.com> Patch / Bug Summary ___________________ Patches : 246 open ( +2) / 2699 closed ( +1) / 2945 total ( +3) Bugs : 770 open ( -1) / 4614 closed (+19) / 5384 total (+18) RFE : 156 open ( -1) / 135 closed ( +1) / 291 total ( +0) New / Reopened Patches ______________________ httplib: allowing stream-type body part in requests (2004-11-12) http://python.org/sf/1065257 opened by Alessandro Forghieri Fix pydoc crashing on unicode strings (2004-11-14) http://python.org/sf/1065986 opened by Cherniavsky Beni Non-ascii in non-unicode __credits__ in Lib/pydoc.py (2004-08-15) http://python.org/sf/1009389 reopened by loewis fix for 1067728: Better handling of float arguments to seek (2004-11-17) http://python.org/sf/1067760 opened by Robert Church Patches Closed ______________ Fix for os.path.expanduser for Win2000 (2002-11-26) http://python.org/sf/643943 closed by jlgijsbers Fix various x86_64 build issues (2004-10-20) http://python.org/sf/1050475 closed by loewis New / Reopened Bugs ___________________ IDLE doesn't start again when saving empty key set (2004-11-11) CLOSED http://python.org/sf/1064535 opened by Hernan Martinez Foffani Parser module asts don't match grammar (2004-11-11) CLOSED http://python.org/sf/1064873 opened by logistix calendar day/month name lookup too slow (2004-11-12) CLOSED http://python.org/sf/1065388 opened by Guido van Rossum sre_parse group limit check missing with 'python -O' (2004-11-12) CLOSED http://python.org/sf/1065427 opened by Sam Rushing pprint.pformat requires __repr__ (2004-11-12) CLOSED http://python.org/sf/1065456 opened by Dave Kuhlman Typo about PyErr_WriteUnraisable() (2004-11-14) http://python.org/sf/1066036 opened by Raise L. Sail datetime.replace method description error (2004-11-14) CLOSED http://python.org/sf/1066438 opened by Brian Underwood special methods become static (2004-11-14) CLOSED http://python.org/sf/1066490 opened by Kevin Quick SegFault in interactive mode when cursor is over a non-ASCII (2004-11-15) http://python.org/sf/1066545 opened by theIgel test_pwd fails on 64bit system (Opteron) (2004-11-15) http://python.org/sf/1066546 opened by Miki Tebeka "Limitations" section of profiler docs is incorrect (2004-11-15) http://python.org/sf/1066607 opened by Nick Coghlan Obsolete info in Tutorial 9.1 (2004-11-15) http://python.org/sf/1067018 opened by Kent Johnson Small typo (2004-11-15) http://python.org/sf/1067023 opened by Scott Miller win32con missing codes VK_VOLUME_MUTE, VK_BROWSER_BACK, ... (2004-11-15) http://python.org/sf/1067153 opened by Jeff Connelly aka shellreef Incorrect length of unicode strings using .encode('utf-8') (2004-11-16) CLOSED http://python.org/sf/1067294 opened by Ed Schofield urllib fails with multiple ftps (2004-11-16) http://python.org/sf/1067702 opened by Russell Owen Overflow error seek()ing with float values > (2 ** 31) - 1 (2004-11-17) http://python.org/sf/1067728 opened by Robert Church wininst --install-script leaves residual files on C: (2004-11-16) http://python.org/sf/1067732 opened by J Livingston Overflow error seek()ing with float values > (2 ** 31) - 1 (2004-11-17) CLOSED http://python.org/sf/1067756 opened by Robert Church Bugs Closed ___________ IDLE aborts on any error when started with "-n" (2004-11-10) http://python.org/sf/1063840 closed by kbk Konqueror can't render docs because of malformed HTML (2004-05-28) http://python.org/sf/962442 closed by fdrake Manual typesets bit-shift operators as guillemet (2004-11-06) http://python.org/sf/1061770 closed by fdrake Fix Decimal's .min() and .max() docs regarding NaNs (2004-11-04) http://python.org/sf/1060644 closed by facundobatista IDLE doesn't start again when saving empty key set (2004-11-11) http://python.org/sf/1064535 closed by kbk Legacy requirements: Incorrect comments? (2004-11-02) http://python.org/sf/1058937 closed by dairiki Parser module asts don't match grammar (2004-11-11) http://python.org/sf/1064873 closed by bcannon optparse docs mildly mangled (2004-11-10) http://python.org/sf/1063757 closed by gward test_subprocess 2.4b2 fails on tru64 (2004-11-10) http://python.org/sf/1063571 closed by astrand difflib HtmlDiff() extra space on inserted 1 character lines (2004-10-26) http://python.org/sf/1054821 closed by tim_one calendar day/month name lookup too slow (2004-11-12) http://python.org/sf/1065388 closed by tim_one sre_parse group limit check missing with 'python -O' (2004-11-12) http://python.org/sf/1065427 closed by effbot pprint.pformat requires __repr__ (2004-11-12) http://python.org/sf/1065456 closed by doerwalter readline()/readlines() (2003-06-27) http://python.org/sf/761787 closed by fullsail datetime.replace method description error (2004-11-14) http://python.org/sf/1066438 closed by tim_one special methods become static (2004-11-15) http://python.org/sf/1066490 closed by mwh pickle won't dump instances after reload (2002-10-25) http://python.org/sf/628925 closed by facundobatista Incorrect length of unicode strings using .encode('utf-8') (2004-11-16) http://python.org/sf/1067294 closed by lemburg IDLE DOES NOT START ON WinXP Pro (2004-08-27) http://python.org/sf/1017978 closed by kbk Overflow error seek()ing with float values > (2 ** 31) - 1 (2004-11-16) http://python.org/sf/1067756 closed by nnorwitz RFE Closed __________ os.run function for convinience and robustness (2004-10-05) http://python.org/sf/1040267 closed by jlgijsbers From sxanth at ceid.upatras.gr Wed Nov 17 11:00:58 2004 From: sxanth at ceid.upatras.gr (Stelios Xanthakis) Date: Wed Nov 17 11:01:01 2004 Subject: [Python-Dev] Re: __pycode__ extension Message-ID: <Pine.GSO.4.61.0411171133160.27454@diogenis.ceid.upatras.gr> Fernando Perez: > Well, not necessarily pushback, but I'd like a clarification at least. > What kind of memory overhead does this introduce? If every function > running around is holding a full copy of all its source, is this > overhead potentially significant? The overhead I'm worried about is the performance cost in the parser. The memory doesn't worry me because computer's memory gets more every year while python source remains compact. It's the debate of maintanability vs optimization. > What happens with decorators, which modify functions but are > not explicit source-level transformations? I don't know about that. I guess decorator declarations are included in the source of a function. > Since this is already fairly straightforward to implement via inspect, > I'd like to see a pretty strong justification for its real need before > seeing it go in. 'inspect' can't keep the source of functions defined with 'exec' or interactively (or can it?). Moreover, you can edit a function at runtime and see it with inspect. As I stated in the OP at c.l.p, this all can be done in python as it is, by using some framework (Zope, whatever). In fact I started from such a framework but then it occured to me that it's the job of the parser to do this. In other words, with __pycode__ python becomes a framework. But I'm not here to sell amway products:) When/if pythoneers decide they like it, let me know. Regards, Stelios From ncoghlan at iinet.net.au Wed Nov 17 11:02:43 2004 From: ncoghlan at iinet.net.au (Nick Coghlan) Date: Wed Nov 17 11:02:48 2004 Subject: [Python-Dev] syntactic shortcut - unpack to variably sized list In-Reply-To: <20041116213506.GB22178@panix.com> References: <000c01c4c838$77bb3da0$0d00a8c0@MAINFRAME> <4195F883.8030309@iinet.net.au> <20041114131708.GA9442@vicky.ecs.soton.ac.uk> <4197B42A.8020401@v.loewis.de> <4198B227.8080309@iinet.net.au> <419920DF.9070002@v.loewis.de> <4199590B.1040908@iinet.net.au> <20041116213506.GB22178@panix.com> Message-ID: <419B21C3.3020700@iinet.net.au> Aahz wrote: > Please fix your Thunderbird to stop adding spurious ^M characters: Sorry about that. It looks like either Google's SMTP server is doing something odd or Thunderbird's TLS implementation has issues - I switched back to my ISP's SMTP server and the problem went away. Cheers, Nick. -- Nick Coghlan | Brisbane, Australia Email: ncoghlan@email.com | Mobile: +61 409 573 268 From ncoghlan at iinet.net.au Wed Nov 17 11:19:08 2004 From: ncoghlan at iinet.net.au (Nick Coghlan) Date: Wed Nov 17 11:19:12 2004 Subject: [Python-Dev] Looking for authoritative documentation on packages, import & ihooks In-Reply-To: <2mu0rpza39.fsf@starship.python.net> References: <864d3709041115075629d15045@mail.gmail.com> <864d37090411160413756ff4c5@mail.gmail.com> <419A52F4.2020408@v.loewis.de> <1f7befae0411161145182e417c@mail.gmail.com> <2mu0rpza39.fsf@starship.python.net> Message-ID: <419B259C.3030007@iinet.net.au> Michael Hudson wrote: >>>Python is open source, so what is wrong with reading import.c? >> >>I've tried. I usually stop when the pool of blood from my eyeballs >>rises above my ankles <0.9 wink>. > > > I'm glad to hear this isn't just me :-) Far from it. . . I was looking around in there when I was implementing the '-m' switch. I *think* I kinda sorta figured out how normal imports usually work (well enough to determine there was not yet a C API equivalent to imp.find_module, anyway). Figuring out what sys.metapath does, on the other hand. . . Cheers, Nick. -- Nick Coghlan | Brisbane, Australia Email: ncoghlan@email.com | Mobile: +61 409 573 268 From ncoghlan at iinet.net.au Wed Nov 17 11:48:22 2004 From: ncoghlan at iinet.net.au (Nick Coghlan) Date: Wed Nov 17 11:48:28 2004 Subject: [Python-Dev] __pycode__ extension In-Reply-To: <419A4C1A.5060502@ceid.upatras.gr> References: <419A4C1A.5060502@ceid.upatras.gr> Message-ID: <419B2C76.1000709@iinet.net.au> Stelios Xanthakis wrote: > > Hi. > > > I posted a message on c.l.p a couple of days ago about a > python patch which adds a member __pycde__ to functions and > classes. This member is a string which holds the python code of > the function/class. (It works in interactively defined code > and exec'd definitions) I think this is a potentially interesting idea, but one worth taking through the PEP process for 2.5. This would have the benefit of clearly identifying the gains provided by the feature, and also identifying potential downsides to be addressed. The number one downside I can see (beyond the extra memory consumed) is the bloating effect on pyc files - for this to work, the compiled bytecode is going to have to contain the original source code as strings. Instead of pyc files being smaller than the originals, they will suddenly be larger (roughly the current size of the py file added to the current size of the pyc file). This may also impact the time required to parse the pyc files at import time - potentially slowing the startup time of the Python interpreter itself, as well as Python applications (a trend in exactly the *wrong* direction). Also, Python does get used in embedded environments, so memory and filesystem space considerations do need to be addressed. One option would be an addition to coflags that is off by default - then the feature could be selectively enabled by an application without impacting normal operation. Cheers, Nick. -0, but willing to be persuaded otherwise (hence, PEP) -- Nick Coghlan | Brisbane, Australia Email: ncoghlan@email.com | Mobile: +61 409 573 268 From sxanth at ceid.upatras.gr Wed Nov 17 13:24:18 2004 From: sxanth at ceid.upatras.gr (Stelios Xanthakis) Date: Wed Nov 17 13:24:22 2004 Subject: [Python-Dev] __pycode__ extension In-Reply-To: <419B2C76.1000709@iinet.net.au> References: <419A4C1A.5060502@ceid.upatras.gr> <419B2C76.1000709@iinet.net.au> Message-ID: <Pine.GSO.4.61.0411171333280.27928@diogenis.ceid.upatras.gr> Hi Nick. > One option would be an addition to coflags that is off by default - then > the feature could be selectively enabled by an application without > impacting normal operation. 100% agree. I used the '-P' command line option to activate this. It is very important to understand that python will run in an entirely different mode. People who are not interested using this should have zero overhead. Only those who are willing to pay the price will get __pycode__ -- and it's not that big. So the real question is: will there be people who will use this? Trying to answer that.. Insight: If this gets in, pythoneers will be tempted to create a file MYSYSTEM.py with one function: #!/usr/bin/python -P def save_world(): F = open ('MYSYSTEM.py') F.write ('#!/usr/bin/python -P\n') for i in globals(): if hasattr (eval(a), '__pycode__'): F.write (eval(a).__pycode__) F.write ('main()') def main(): import code code.interact ('', raw_input, globals()) And that will give birth to their own personal incremental python shell/library. > The number one downside I can see (beyond the extra memory consumed) is > the bloating effect on pyc files - for this to work, the compiled > bytecode is going to have to contain the original source code as > strings. The __pycode__ attribute is not marshaled. The logic behind this is that if we edit a function/class and save it in a .pyc file, the next time the .py module is executed it will overwrite the .pyc file and our changes are lost. So for 'import'ed code, __pycode__=None Hm? > I think this is a potentially interesting idea, but one worth taking > through the PEP process for 2.5. This would have the benefit of clearly > identifying the gains provided by the feature, and also identifying > potential downsides to be addressed. Definitelly. But before starting a PEP I'd like to gather feedback and ideas, have a good patch vs 2.4, and, well, understand python source a little better. Stelios From anthony at interlink.com.au Wed Nov 17 15:59:36 2004 From: anthony at interlink.com.au (Anthony Baxter) Date: Wed Nov 17 15:59:47 2004 Subject: [Python-Dev] TRUNK FROZEN for 2.4rc1 from around 1200 UTC on Thurs 18 Message-ID: <200411171459.iAHExag25031@burswood.off.ekorp.com> Please hold off on any checkins after 1200 UTC on Thursday 18th (about 22 hours after this message was sent). We're cutting 2.4rc1 then. Assuming all goes well, we'll be looking at a 2.4 final for Nov 30. Could people please be _very_ conservative with checkins between now and 2.4 final? A brown-paper-bag 2.4.1 would suck :-( ta Anthony From pje at telecommunity.com Wed Nov 17 17:01:56 2004 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed Nov 17 17:00:58 2004 Subject: [Python-Dev] __pycode__ extension In-Reply-To: <Pine.GSO.4.61.0411171333280.27928@diogenis.ceid.upatras.gr > References: <419B2C76.1000709@iinet.net.au> <419A4C1A.5060502@ceid.upatras.gr> <419B2C76.1000709@iinet.net.au> Message-ID: <5.1.1.6.0.20041117105731.020e4230@mail.telecommunity.com> At 02:24 PM 11/17/04 +0200, Stelios Xanthakis wrote: > #!/usr/bin/python -P > def save_world(): > F = open ('MYSYSTEM.py') > F.write ('#!/usr/bin/python -P\n') > for i in globals(): > if hasattr (eval(a), '__pycode__'): > F.write (eval(a).__pycode__) There are bugs in this code. First, the wrong variable is used for eval(), and second, it shouldn't use eval. If there's a global named 'F', for example, the code above will not save it, because it will see the local 'F' instead. Loop over globals().values(), and do not use eval. >The __pycode__ attribute is not marshaled. >The logic behind this is that if we edit a function/class >and save it in a .pyc file, the next time the .py module >is executed it will overwrite the .pyc file and our changes >are lost. >So for 'import'ed code, __pycode__=None > >Hm? What good is that? Likewise, I don't see the point of having this only enabled when a command-line option is given. If the idea is just to allow saving code from interactive mode, why not just modify the interactive mode to do this? From FBatista at uniFON.com.ar Wed Nov 17 18:52:32 2004 From: FBatista at uniFON.com.ar (Batista, Facundo) Date: Wed Nov 17 18:54:47 2004 Subject: [Python-Dev] Py 2.4 b2 announcement Message-ID: <A128D751272CD411BC9200508BC2194D053C7C09@escpl.tcp.com.ar> Is some weird cache issue from my side, or Py2.4b2 is not announced at http://www.python.org/? Regards, Facundo Batista Desarrollo de Red fbatista@unifon.com.ar (54 11) 5130-4643 Cel: 15 5097 5024 From amk at amk.ca Wed Nov 17 19:00:29 2004 From: amk at amk.ca (A.M. Kuchling) Date: Wed Nov 17 19:04:20 2004 Subject: [Python-Dev] Py 2.4 b2 announcement In-Reply-To: <A128D751272CD411BC9200508BC2194D053C7C09@escpl.tcp.com.ar> References: <A128D751272CD411BC9200508BC2194D053C7C09@escpl.tcp.com.ar> Message-ID: <20041117180029.GA4501@rogue.amk.ca> On Wed, Nov 17, 2004 at 02:52:32PM -0300, Batista, Facundo wrote: > Is some weird cache issue from my side, or Py2.4b2 is not announced at > http://www.python.org/? You're correct; looking at the CVS log for index.ht, there's an entry for 2.4b1 but not b2. I've updated the page. --amk From pedronis at bluewin.ch Wed Nov 17 20:37:18 2004 From: pedronis at bluewin.ch (Samuele Pedroni) Date: Wed Nov 17 20:34:14 2004 Subject: [Python-Dev] __pycode__ extension In-Reply-To: <5.1.1.6.0.20041117105731.020e4230@mail.telecommunity.com> References: <419B2C76.1000709@iinet.net.au> <419A4C1A.5060502@ceid.upatras.gr> <419B2C76.1000709@iinet.net.au> <5.1.1.6.0.20041117105731.020e4230@mail.telecommunity.com> Message-ID: <419BA86E.70309@bluewin.ch> Phillip J. Eby wrote: > At 02:24 PM 11/17/04 +0200, Stelios Xanthakis wrote: > >> #!/usr/bin/python -P >> def save_world(): >> F = open ('MYSYSTEM.py') >> F.write ('#!/usr/bin/python -P\n') >> for i in globals(): >> if hasattr (eval(a), '__pycode__'): >> F.write (eval(a).__pycode__) > > > There are bugs in this code. First, the wrong variable is used for > eval(), and second, it shouldn't use eval. If there's a global named > 'F', for example, the code above will not save it, because it will see > the local 'F' instead. Loop over globals().values(), and do not use eval. > > >> The __pycode__ attribute is not marshaled. >> The logic behind this is that if we edit a function/class >> and save it in a .pyc file, the next time the .py module >> is executed it will overwrite the .pyc file and our changes >> are lost. >> So for 'import'ed code, __pycode__=None >> >> Hm? > > > What good is that? I agree with the spirit of the observation, but I think that aspect is immaterial. Personally I think some form of this feature is valuable as long as it means getting inspect.getsource (and companions) or some variation working no matter what. So as long as what goes in the attribute covers the cases not working today, that should be fine. > Likewise, I don't see the point of having this only > enabled when a command-line option is given. well, I think some people may bring memory consumption considerations to the table depending of what the final implementation is. For embedding scenarios I suspect that a build time define could be ok. But I think we should rember that some people today rely on the fact that shipping only pycs files means not shipping directly the source. From mal at egenix.com Wed Nov 17 20:39:22 2004 From: mal at egenix.com (M.-A. Lemburg) Date: Wed Nov 17 20:39:26 2004 Subject: [Python-Dev] sys.exit(47) ? Message-ID: <419BA8EA.7030002@egenix.com> Hi there, something in the regression suite is generating a temporary file with content: #!/bin/sh exec /path/to/python -c 'import sys; sys.exit(47)' and not cleaning up after itself. This started around Oct. 13. Looking at the regression suite test_subprocess.py seems to be the cause. BTW, wouldn't sys.exit(42) be more appropriate :-? -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Nov 17 2004) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From pedronis at bluewin.ch Wed Nov 17 20:47:42 2004 From: pedronis at bluewin.ch (Samuele Pedroni) Date: Wed Nov 17 20:44:59 2004 Subject: [Python-Dev] __pycode__ extension In-Reply-To: <419BA86E.70309@bluewin.ch> References: <419B2C76.1000709@iinet.net.au> <419A4C1A.5060502@ceid.upatras.gr> <419B2C76.1000709@iinet.net.au> <5.1.1.6.0.20041117105731.020e4230@mail.telecommunity.com> <419BA86E.70309@bluewin.ch> Message-ID: <419BAADE.4090800@bluewin.ch> Samuele Pedroni wrote: >> What good is that? > > > > I agree with the spirit of the observation, but I think that aspect is > immaterial. Personally I think some form of this feature is valuable as > long as it means getting inspect.getsource (and companions) or some > variation working no matter what. So as long as what goes in the > attribute covers the cases not working today, that should be fine. > of course depending on the scenarios one has in mind, working may mean produce always an output but with the same limitations from code in py files, or produce a correct output even in the case of file editings. From astrand at lysator.liu.se Wed Nov 17 20:53:55 2004 From: astrand at lysator.liu.se (Peter Astrand) Date: Wed Nov 17 20:54:06 2004 Subject: [Python-Dev] Re: subprocess and EINTR errnos In-Reply-To: <20041111034803.GA16175@isc.upenn.edu> References: <20041111034803.GA16175@isc.upenn.edu> Message-ID: <Pine.GSO.4.51L2.0411162121580.11957@koeberg.lysator.liu.se> On Wed, 10 Nov 2004, John P Speno wrote: Hi, sorry for the delayed response. > While using subprocess (aka popen5), I came across one potential gotcha. I've had > exceptions ending like this: > > File "test.py", line 5, in test > cmd = popen5.Popen(args, stdout=PIPE) > File "popen5.py", line 577, in __init__ > data = os.read(errpipe_read, 1048576) # Exceptions limited to 1 MB > OSError: [Errno 4] Interrupted system call > > (on Solaris 9) > > Would it make sense for subprocess to use a more robust read() function > which can handle these cases, i.e. when the parent's read on the pipe > to the child's stderr is interrupted by a system call, and returns EINTR? > I imagine it could catch EINTR and EAGAIN and retry the failed read(). I assume you are using signals in your application? The os.read above is not the only system call that can fail with EINTR. subprocess.py is full of other system calls that can fail, and I suspect that many other Python modules are as well. I've made a patch (attached) to subprocess.py (and test_subprocess.py) that should guard against EINTR, but I haven't committed it yet. It's quite large. Are Python modules supposed to handle EINTR? Why not let the C code handle this? Or, perhaps the signal module should provide a sigaction function, so that users can use SA_RESTART. Index: subprocess.py =================================================================== RCS file: /cvsroot/python/python/dist/src/Lib/subprocess.py,v retrieving revision 1.8 diff -u -r1.8 subprocess.py --- subprocess.py 7 Nov 2004 14:30:34 -0000 1.8 +++ subprocess.py 17 Nov 2004 19:42:30 -0000 @@ -888,6 +888,50 @@ pass + def _read_no_intr(self, fd, buffersize): + """Like os.read, but retries on EINTR""" + while True: + try: + return os.read(fd, buffersize) + except OSError, e: + if e.errno == errno.EINTR: + continue + else: + raise + + + def _read_all(self, fd, buffersize): + """Like os.read, but retries on EINTR, and reads until EOF""" + all = "" + while True: + data = self._read_no_intr(fd, buffersize) + all += data + if data == "": + return all + + + def _write_no_intr(self, fd, s): + """Like os.write, but retries on EINTR""" + while True: + try: + return os.write(fd, s) + except OSError, e: + if e.errno == errno.EINTR: + continue + else: + raise + + def _waitpid_no_intr(self, pid, options): + """Like os.waitpid, but retries on EINTR""" + while True: + try: + return os.waitpid(pid, options) + except OSError, e: + if e.errno == errno.EINTR: + continue + else: + raise + def _execute_child(self, args, executable, preexec_fn, close_fds, cwd, env, universal_newlines, startupinfo, creationflags, shell, @@ -963,7 +1007,7 @@ exc_value, tb) exc_value.child_traceback = ''.join(exc_lines) - os.write(errpipe_write, pickle.dumps(exc_value)) + self._write_no_intr(errpipe_write, pickle.dumps(exc_value)) # This exitcode won't be reported to applications, so it # really doesn't matter what we return. @@ -979,7 +1023,7 @@ os.close(errwrite) # Wait for exec to fail or succeed; possibly raising exception - data = os.read(errpipe_read, 1048576) # Exceptions limited to 1 MB + data = self._read_all(errpipe_read, 1048576) # Exceptions limited to 1 MB os.close(errpipe_read) if data != "": child_exception = pickle.loads(data) @@ -1003,7 +1047,7 @@ attribute.""" if self.returncode == None: try: - pid, sts = os.waitpid(self.pid, os.WNOHANG) + pid, sts = self._waitpid_no_intr(self.pid, os.WNOHANG) if pid == self.pid: self._handle_exitstatus(sts) except os.error: @@ -1015,7 +1059,7 @@ """Wait for child process to terminate. Returns returncode attribute.""" if self.returncode == None: - pid, sts = os.waitpid(self.pid, 0) + pid, sts = self._waitpid_no_intr(self.pid, 0) self._handle_exitstatus(sts) return self.returncode @@ -1049,27 +1093,33 @@ stderr = [] while read_set or write_set: - rlist, wlist, xlist = select.select(read_set, write_set, []) + try: + rlist, wlist, xlist = select.select(read_set, write_set, []) + except select.error, e: + if e[0] == errno.EINTR: + continue + else: + raise if self.stdin in wlist: # When select has indicated that the file is writable, # we can write up to PIPE_BUF bytes without risk # blocking. POSIX defines PIPE_BUF >= 512 - bytes_written = os.write(self.stdin.fileno(), input[:512]) + bytes_written = self._write_no_intr(self.stdin.fileno(), input[:512]) input = input[bytes_written:] if not input: self.stdin.close() write_set.remove(self.stdin) if self.stdout in rlist: - data = os.read(self.stdout.fileno(), 1024) + data = self._read_no_intr(self.stdout.fileno(), 1024) if data == "": self.stdout.close() read_set.remove(self.stdout) stdout.append(data) if self.stderr in rlist: - data = os.read(self.stderr.fileno(), 1024) + data = self._read_no_intr(self.stderr.fileno(), 1024) if data == "": self.stderr.close() read_set.remove(self.stderr) Index: test/test_subprocess.py =================================================================== RCS file: /cvsroot/python/python/dist/src/Lib/test/test_subprocess.py,v retrieving revision 1.14 diff -u -r1.14 test_subprocess.py --- test/test_subprocess.py 12 Nov 2004 15:51:48 -0000 1.14 +++ test/test_subprocess.py 17 Nov 2004 19:42:30 -0000 @@ -7,6 +7,7 @@ import tempfile import time import re +import errno mswindows = (sys.platform == "win32") @@ -35,6 +36,16 @@ fname = tempfile.mktemp() return os.open(fname, os.O_RDWR|os.O_CREAT), fname + def read_no_intr(self, obj): + while True: + try: + return obj.read() + except IOError, e: + if e.errno == errno.EINTR: + continue + else: + raise + # # Generic tests # @@ -123,7 +134,7 @@ p = subprocess.Popen([sys.executable, "-c", 'import sys; sys.stdout.write("orange")'], stdout=subprocess.PIPE) - self.assertEqual(p.stdout.read(), "orange") + self.assertEqual(self.read_no_intr(p.stdout), "orange") def test_stdout_filedes(self): # stdout is set to open file descriptor @@ -151,7 +162,7 @@ p = subprocess.Popen([sys.executable, "-c", 'import sys; sys.stderr.write("strawberry")'], stderr=subprocess.PIPE) - self.assertEqual(remove_stderr_debug_decorations(p.stderr.read()), + self.assertEqual(remove_stderr_debug_decorations(self.read_no_intr(p.stderr)), "strawberry") def test_stderr_filedes(self): @@ -186,7 +197,7 @@ 'sys.stderr.write("orange")'], stdout=subprocess.PIPE, stderr=subprocess.STDOUT) - output = p.stdout.read() + output = self.read_no_intr(p.stdout) stripped = remove_stderr_debug_decorations(output) self.assertEqual(stripped, "appleorange") @@ -220,7 +231,7 @@ stdout=subprocess.PIPE, cwd=tmpdir) normcase = os.path.normcase - self.assertEqual(normcase(p.stdout.read()), normcase(tmpdir)) + self.assertEqual(normcase(self.read_no_intr(p.stdout)), normcase(tmpdir)) def test_env(self): newenv = os.environ.copy() @@ -230,7 +241,7 @@ 'sys.stdout.write(os.getenv("FRUIT"))'], stdout=subprocess.PIPE, env=newenv) - self.assertEqual(p.stdout.read(), "orange") + self.assertEqual(self.read_no_intr(p.stdout), "orange") def test_communicate(self): p = subprocess.Popen([sys.executable, "-c", @@ -305,7 +316,8 @@ 'sys.stdout.write("\\nline6");'], stdout=subprocess.PIPE, universal_newlines=1) - stdout = p.stdout.read() + + stdout = self.read_no_intr(p.stdout) if hasattr(open, 'newlines'): # Interpreter with universal newline support self.assertEqual(stdout, @@ -343,7 +355,7 @@ def test_no_leaking(self): # Make sure we leak no resources - max_handles = 1026 # too much for most UNIX systems + max_handles = 10 # too much for most UNIX systems if mswindows: max_handles = 65 # a full test is too slow on Windows for i in range(max_handles): @@ -424,7 +436,7 @@ 'sys.stdout.write(os.getenv("FRUIT"))'], stdout=subprocess.PIPE, preexec_fn=lambda: os.putenv("FRUIT", "apple")) - self.assertEqual(p.stdout.read(), "apple") + self.assertEqual(self.read_no_intr(p.stdout), "apple") def test_args_string(self): # args is a string @@ -457,7 +469,7 @@ p = subprocess.Popen(["echo $FRUIT"], shell=1, stdout=subprocess.PIPE, env=newenv) - self.assertEqual(p.stdout.read().strip(), "apple") + self.assertEqual(self.read_no_intr(p.stdout).strip(), "apple") def test_shell_string(self): # Run command through the shell (string) @@ -466,7 +478,7 @@ p = subprocess.Popen("echo $FRUIT", shell=1, stdout=subprocess.PIPE, env=newenv) - self.assertEqual(p.stdout.read().strip(), "apple") + self.assertEqual(self.read_no_intr(p.stdout).strip(), "apple") def test_call_string(self): # call() function with string argument on UNIX @@ -525,7 +537,7 @@ p = subprocess.Popen(["set"], shell=1, stdout=subprocess.PIPE, env=newenv) - self.assertNotEqual(p.stdout.read().find("physalis"), -1) + self.assertNotEqual(self.read_no_intr(p.stdout).find("physalis"), -1) def test_shell_string(self): # Run command through the shell (string) @@ -534,7 +546,7 @@ p = subprocess.Popen("set", shell=1, stdout=subprocess.PIPE, env=newenv) - self.assertNotEqual(p.stdout.read().find("physalis"), -1) + self.assertNotEqual(self.read_no_intr(p.stdout).find("physalis"), -1) def test_call_string(self): # call() function with string argument on Windows /Peter ?strand <astrand@lysator.liu.se> From fperez528 at yahoo.com Wed Nov 17 20:56:56 2004 From: fperez528 at yahoo.com (Fernando Perez) Date: Wed Nov 17 20:57:01 2004 Subject: [Python-Dev] Re: __pycode__ extension References: <419B2C76.1000709@iinet.net.au> <419A4C1A.5060502@ceid.upatras.gr> <5.1.1.6.0.20041117105731.020e4230@mail.telecommunity.com> Message-ID: <cngae8$s80$1@sea.gmane.org> Phillip J. Eby wrote: > If the idea is just to allow saving code from interactive mode, why not > just modify the interactive mode to do this? IPython already has most of this: In [3]: os.path.walk?? Type: function Base Class: <type 'function'> String Form: <function walk at 0x550529cc> Namespace: Interactive File: /usr/src/build/394694-i386/install/usr/lib/python2.3/posixpath.py Definition: os.path.walk(top, func, arg) Source: def walk(top, func, arg): """Directory tree walk with callback function. For each directory in the directory tree rooted at top (including top itself, but excluding '.' and '..'), call func(arg, dirname, fnames). dirname is the name of the directory, and fnames a list of the names of the files and subdirectories in dirname (excluding '.' and '..'). func may modify the fnames list in-place (e.g. via del or slice assignment), and walk will only recurse into the subdirectories whose names remain in fnames; this can be used to implement a filter, or to impose a specific order of visiting. No semantics are defined for, or required of, arg, beyond that arg is always passed to func. It can be used, e.g., to pass a filename pattern, or a mutable object designed to accumulate statistics. Passing None for arg is common.""" try: names = os.listdir(top) except os.error: return func(arg, top, names) for name in names: name = join(top, name) try: st = os.lstat(name) except os.error: continue if stat.S_ISDIR(st.st_mode): walk(name, func, arg) Since it's already available (anyone can copy the code if they want, it's BSD), I am very weary of the bloat this would introduce if it were a default feature. So I'd like to see a good argument about the footprint increase of this proposal. Best, f From bac at OCF.Berkeley.EDU Wed Nov 17 20:57:36 2004 From: bac at OCF.Berkeley.EDU (Brett C.) Date: Wed Nov 17 20:57:50 2004 Subject: [Python-Dev] proposal+patch: sys.gettickeraccumulation() In-Reply-To: <41995FBF.3080207@iinet.net.au> References: <200411152028.iAFKSJgq040962@boa.lbl.gov> <41995FBF.3080207@iinet.net.au> Message-ID: <419BAD30.1010200@ocf.berkeley.edu> Nick Coghlan wrote: > Ralf W. Grosse-Kunstleve wrote: > >>As we work on new algorithms we can easily monitor the time/tick >>without a runtime penalty. As long as the ratio stays reasonably high >>(e.g. because we use numarray-like array operations already coded in >>C++) we don't have to do anything. If the ratio goes down significantly >>we can run the profiler to do detailed analysis of the new code. > > > That seems like a reasonable approach to me - one could easily make use > of the ticker information in a test framework. So I'll withdraw my > objection and give the new function a +0. > I have been mulling over this proposal and I think I am finally settling on +0 as long as it can be shown that it does not hurt performance at all (yes, it shouldn't but this is the eval loop we are talking about and that thing is a touchy beast). This feels like a move towards providing the same level of infor that Java's JMX provides (which I am not saying is good or bad; just been writing a Python vs. Java research paper and so I have partial Java-on-the-brain). I think the trick is going to be making it clear in the docs that this is meant to give people a rough idea of where there is still a lot of Python code being used compared to C and not and should not be used to measure performance. -Brett From astrand at lysator.liu.se Wed Nov 17 21:13:26 2004 From: astrand at lysator.liu.se (Peter Astrand) Date: Wed Nov 17 21:14:11 2004 Subject: [Python-Dev] sys.exit(47) ? In-Reply-To: <419BA8EA.7030002@egenix.com> References: <419BA8EA.7030002@egenix.com> Message-ID: <Pine.GSO.4.51L2.0411172107120.2244@koeberg.lysator.liu.se> On Wed, 17 Nov 2004, M.-A. Lemburg wrote: > something in the regression suite is generating a temporary > file with content: > > #!/bin/sh > exec /path/to/python -c 'import sys; sys.exit(47)' > > and not cleaning up after itself. > > This started around Oct. 13. Looking at the regression suite > test_subprocess.py seems to be the cause. Thanks for pointing this out. Now fixed, in revision 1.15 of test_subprocess.py. > BTW, wouldn't sys.exit(42) be more appropriate :-? 42 is not as random :) See http://en.wikipedia.org/wiki/47_%28number%29 /Peter ?strand <astrand@lysator.liu.se> From foom at fuhm.net Wed Nov 17 21:29:03 2004 From: foom at fuhm.net (James Y Knight) Date: Wed Nov 17 21:29:06 2004 Subject: [Python-Dev] Re: subprocess and EINTR errnos In-Reply-To: <Pine.GSO.4.51L2.0411162121580.11957@koeberg.lysator.liu.se> References: <20041111034803.GA16175@isc.upenn.edu> <Pine.GSO.4.51L2.0411162121580.11957@koeberg.lysator.liu.se> Message-ID: <52765627-38D7-11D9-AA57-000A95A50FB2@fuhm.net> On Nov 17, 2004, at 2:53 PM, Peter Astrand wrote: > I assume you are using signals in your application? The os.read above > is > not the only system call that can fail with EINTR. subprocess.py is > full > of other system calls that can fail, and I suspect that many other > Python > modules are as well. > > I've made a patch (attached) to subprocess.py (and test_subprocess.py) > that should guard against EINTR, but I haven't committed it yet. It's > quite large. > > Are Python modules supposed to handle EINTR? Why not let the C code > handle > this? Or, perhaps the signal module should provide a sigaction > function, > so that users can use SA_RESTART. Here's a simple way to demonstrate the same problem (without subprocess). I think any solution should not be at the subprocess.py level. >>> import os,sys,signal >>> signal.signal(signal.SIGCHLD, lambda x,y: sys.stderr.write("SIGCHLD!\n")) 0 >>> f=os.popen("sleep 5; echo 'Foo'"); print f.read() SIGCHLD! Traceback (most recent call last): File "<stdin>", line 1, in ? IOError: [Errno 4] Interrupted system call This is an issue that Twisted has run into. Because Twisted installs its own SIGCHLD handler, other code which uses popen and then reads/writes the pipe *will* fail, because the SIGCHLD interrupts the final read. The proposed solution for twisted (which no one has implemented yet) is to make a small C module which simply adds the SA_RESTART flag to an installed signal handler. This has a disadvantage: because python queues up signals to run during normal interpreter execution, NOT at interrupt time, the handler won't be called until after the read finally returns. In our particular case, that's good enough, as the SIGCHLD handler is not particularly timing critical. As long as it gets called at some point, that's fine. However, a more general solution would be to emulate the SA_RESTART functionality at Python level: - Add an optional argument to signal.signal to specify whether you want it to restart interrupted calls after handling the signal. This would *not* set SA_RESTART, but would instead set a flag for the python read/write/etc wrappers to look at. - Make the python wrapper for read/write/etc handle EINTR internally: first dispatch to the waiting signal handler, then call the syscall again, if the sighandler was installed with the restart argument. From my documentation for SA_RESTART: """The affected system calls include open(2), read(2), write(2), sendto(2), recvfrom(2), sendmsg(2) and recvmsg(2) on a communications channel or a slow device (such as a terminal, but not a regular file) and during a wait(2) or ioctl(2).""" So, to emulate SA_RESTART, all those should be wrapped to restart in appropriate conditions. James From martin at v.loewis.de Wed Nov 17 21:30:43 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed Nov 17 21:30:48 2004 Subject: [Python-Dev] os.access versus os.exist In-Reply-To: <3A81C87DC164034AA4E2DDFE11D258E3245214@exchange.hqamor.amorhq.net> References: <3A81C87DC164034AA4E2DDFE11D258E3245214@exchange.hqamor.amorhq.net> Message-ID: <419BB4F3.4040402@v.loewis.de> Robert Brewer wrote: > Completely understood about LocalSystem. But IMO, os.stat should > raise a more specific and correct error. As you pointed out, calling > open() raises "IOError: [Errno 13] Permission denied"; it would be > nice if stat did the same (in my example). The platform-specific API > call (for example, Windows _stati64) doesn't provide this--it would > require another test inside posix_do_stat to return a more > appropriate error. But it would be a nice touch for us end-users. No. Python faking up errors that the platform doesn't provide is evil. You are entitled to precisely the error that the platform reported. Now, MSVC stat() is severely constraint, in several ways (e.g. it doesn't support sub-second time-stamps, either), so rewriting stat with plain Win32 API might be reasonable, at which point giving more precise error would be an option. > In my opinion, "no such file" is the wrong error message and is > therefore a bug. But I can accept others disagreeing, since os.stat() > simply wraps Windows' _stat* calls (at least, in my example). It sure looks like a bug - but one of msvcrt.dll. Regards, Martin From pje at telecommunity.com Wed Nov 17 21:34:08 2004 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed Nov 17 21:33:09 2004 Subject: [Python-Dev] Re: __pycode__ extension In-Reply-To: <cngae8$s80$1@sea.gmane.org> References: <419B2C76.1000709@iinet.net.au> <419A4C1A.5060502@ceid.upatras.gr> <5.1.1.6.0.20041117105731.020e4230@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20041117153303.02c412f0@mail.telecommunity.com> At 12:56 PM 11/17/04 -0700, Fernando Perez wrote: >Phillip J. Eby wrote: > > > If the idea is just to allow saving code from interactive mode, why not > > just modify the interactive mode to do this? > >IPython already has most of this: > >In [3]: os.path.walk?? >Type: function >Base Class: <type 'function'> >String Form: <function walk at 0x550529cc> >Namespace: Interactive >File: >/usr/src/build/394694-i386/install/usr/lib/python2.3/posixpath.py If I understand correctly, this is unrelated. The OP was about retaining source of functions defined in interactive mode, not the source of functions from files. 'os.path.walk' was obviously not typed in using interactive mode. :) From mal at egenix.com Wed Nov 17 21:41:16 2004 From: mal at egenix.com (M.-A. Lemburg) Date: Wed Nov 17 21:41:18 2004 Subject: [Python-Dev] sys.exit(47) ? In-Reply-To: <Pine.GSO.4.51L2.0411172107120.2244@koeberg.lysator.liu.se> References: <419BA8EA.7030002@egenix.com> <Pine.GSO.4.51L2.0411172107120.2244@koeberg.lysator.liu.se> Message-ID: <419BB76C.1020102@egenix.com> Peter Astrand wrote: > On Wed, 17 Nov 2004, M.-A. Lemburg wrote: > > >>something in the regression suite is generating a temporary >>file with content: >> >>#!/bin/sh >>exec /path/to/python -c 'import sys; sys.exit(47)' >> >>and not cleaning up after itself. >> >>This started around Oct. 13. Looking at the regression suite >>test_subprocess.py seems to be the cause. > > > Thanks for pointing this out. Now fixed, in revision 1.15 of > test_subprocess.py. Thanks, I'll see whether my nightly build script will pick up the change. >>BTW, wouldn't sys.exit(42) be more appropriate :-? > > > 42 is not as random :) See http://en.wikipedia.org/wiki/47_%28number%29 Interesting reference :-) http://en.wikipedia.org/wiki/42_%28number%29 AFAIR, Tim Peters introduced that number to the Python universe. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Nov 17 2004) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From fumanchu at amor.org Wed Nov 17 22:08:59 2004 From: fumanchu at amor.org (Robert Brewer) Date: Wed Nov 17 22:10:35 2004 Subject: [Python-Dev] os.access versus os.exist Message-ID: <3A81C87DC164034AA4E2DDFE11D258E324522B@exchange.hqamor.amorhq.net> "Martin v. L?wis" wrote: > Sent: Wednesday, November 17, 2004 12:31 PM > To: Robert Brewer > Cc: python-dev@python.org > Subject: Re: [Python-Dev] os.access versus os.exist > > > Robert Brewer wrote: > > Completely understood about LocalSystem. But IMO, os.stat should > > raise a more specific and correct error. As you pointed out, calling > > open() raises "IOError: [Errno 13] Permission denied"; it would be > > nice if stat did the same (in my example). The platform-specific API > > call (for example, Windows _stati64) doesn't provide this--it would > > require another test inside posix_do_stat to return a more > > appropriate error. But it would be a nice touch for us end-users. > > No. Python faking up errors that the platform doesn't provide is evil. > You are entitled to precisely the error that the platform reported. > > Now, MSVC stat() is severely constraint, in several ways (e.g. it > doesn't support sub-second time-stamps, either), so rewriting stat > with plain Win32 API might be reasonable, at which point giving > more precise error would be an option. > > > In my opinion, "no such file" is the wrong error message and is > > therefore a bug. But I can accept others disagreeing, since > os.stat() > > simply wraps Windows' _stat* calls (at least, in my example). > > It sure looks like a bug - but one of msvcrt.dll. OK. I'll file a small doc patch for os.path.exists, then, I think. Thanks! Robert Brewer MIS Amor Ministries fumanchu@amor.org From fperez528 at yahoo.com Wed Nov 17 22:35:31 2004 From: fperez528 at yahoo.com (Fernando Perez) Date: Wed Nov 17 22:35:39 2004 Subject: [Python-Dev] Re: Re: __pycode__ extension References: <419B2C76.1000709@iinet.net.au> <419A4C1A.5060502@ceid.upatras.gr> <5.1.1.6.0.20041117105731.020e4230@mail.telecommunity.com> <cngae8$s80$1@sea.gmane.org> <5.1.1.6.0.20041117153303.02c412f0@mail.telecommunity.com> Message-ID: <cngg73$g40$1@sea.gmane.org> Phillip J. Eby wrote: > If I understand correctly, this is unrelated. The OP was about retaining > source of functions defined in interactive mode, not the source of > functions from files. 'os.path.walk' was obviously not typed in using > interactive mode. :) OK, my misunderstanding. You are right, and inspect does NOT seem to provide a way to access the source of interactively defined functions (at least not that I've seen): In [6]: def foo(x): ...: print x ...: In [7]: psource foo No source found for foo Sorry for the confusion. best, f From barry at python.org Wed Nov 17 23:19:25 2004 From: barry at python.org (Barry Warsaw) Date: Wed Nov 17 23:19:31 2004 Subject: [Python-Dev] __pycode__ extension In-Reply-To: <419BA86E.70309@bluewin.ch> References: <419B2C76.1000709@iinet.net.au> <419A4C1A.5060502@ceid.upatras.gr> <419B2C76.1000709@iinet.net.au> <5.1.1.6.0.20041117105731.020e4230@mail.telecommunity.com> <419BA86E.70309@bluewin.ch> Message-ID: <1100729964.20177.29.camel@geddy.wooz.org> On Wed, 2004-11-17 at 14:37, Samuele Pedroni wrote: > But I think we should rember that some people today rely on the fact > that shipping only pycs files means not shipping directly the source. Absolutely. If this is added (and in general it sounds like a decent, though PEP-worthy idea), there must be a way to turn it off. I'm not sure a build-time configuration option is enough, but a command line option might be. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 307 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-dev/attachments/20041117/e423104b/attachment.pgp From pedronis at bluewin.ch Thu Nov 18 00:44:56 2004 From: pedronis at bluewin.ch (Samuele Pedroni) Date: Thu Nov 18 00:41:26 2004 Subject: [Python-Dev] __pycode__ extension In-Reply-To: <1100729964.20177.29.camel@geddy.wooz.org> References: <419B2C76.1000709@iinet.net.au> <419A4C1A.5060502@ceid.upatras.gr> <419B2C76.1000709@iinet.net.au> <5.1.1.6.0.20041117105731.020e4230@mail.telecommunity.com> <419BA86E.70309@bluewin.ch> <1100729964.20177.29.camel@geddy.wooz.org> Message-ID: <419BE278.7030402@bluewin.ch> Barry Warsaw wrote: > On Wed, 2004-11-17 at 14:37, Samuele Pedroni wrote: > > >>But I think we should rember that some people today rely on the fact >>that shipping only pycs files means not shipping directly the source. > > > Absolutely. yes but further thinking I can imagine the following tension - people will still want a way to ship without shipping the source, or such that reconstructing the source is not too easy, with varying degrees of what is considered enough "secure" - although this can already be done partly today, this feature will make more tempting (especially in combination with features that can be expected from/constructed on top of the ast-branch or with the compiler package) to write e.g decorators that operate on functions by playing with their source. But then these tools will only work if the source or asts are there at runtime in some form [bytecode manipulation can be used similary but is less portable across python implementation, and seemingly harder than ast manipulations] > If this is added (and in general it sounds like a decent, > though PEP-worthy idea), there must be a way to turn it off. I'm not > sure a build-time configuration option is enough, but a command line > option might be. > and/or some setting changeable through a sys method As I implicitly said I can see (at least these) three levels for the feature 1) disabled 2) store only the sources that cannot be recovered accessing .py files, i.e. make inspect.getsource always work up to editing of the .py files 3) make things robust wrt .py files editing and other corner cases, i.e. likely store everything to some extent 3 can be implemented without storing things in pyc files, as long as there are .py files around by an approach reading both in parallel OTOH to have the tools described above work in a pyc-only scenario, source or asts would need to be stored in the pycs themself, opening the question about the resulting sufficient obfuscation/expressivity tradeoff From carribeiro at gmail.com Thu Nov 18 03:43:39 2004 From: carribeiro at gmail.com (Carlos Ribeiro) Date: Thu Nov 18 03:44:24 2004 Subject: [Python-Dev] syntactic shortcut - unpack to variably sized list In-Reply-To: <000c01c4c838$77bb3da0$0d00a8c0@MAINFRAME> References: <000c01c4c838$77bb3da0$0d00a8c0@MAINFRAME> Message-ID: <864d3709041117184370f63c1b@mail.gmail.com> On Thu, 11 Nov 2004 22:50:21 +0100, Johan Hahn <johahn@home.se> wrote: > Hi > > As far as I can tell from the archive, this has not been discussed before. > This is the second time in less than a week that I have stumbled over the rather > clumsy syntax of extracting some elements of a sequence and at the same time > remove those from the sequence: > >>> L = 'a b 1 2 3'.split(' ') > >>> a,b,L = L[0], L[1], L[2:] I am really late on this thread, but anyway, I've come up with another approach to solve the problem using iterators. It uses iterator that is guaranteed to always return a fixed number of elements, regardless of the size of the sequence; when it finishes, it returns the tail of the sequence as the last argument. This is a simple-minded proof of concept, and it's surely highly optimizable in at least a hundred different ways :-) def iunpack(seq, times, defaultitem=None): for i in range(times): if i < len(seq): yield seq[i] else: yield defaultitem if i < len(seq): yield seq[i+1:] else: yield () Usage is as follows: >>> tuple(iunpack((1,2,3,4), 1)) (1, (2, 3, 4)) >>> tuple(iunpack((1,2,3,4), 2)) (1, 2, (3, 4)) >>> tuple(iunpack((1,2,3,4), 4)) (1, 2, 3, 4, ()) >>> tuple(iunpack((1,2,3,4), 6)) (1, 2, 3, 4, None, None, ()) As it is, it fails if the requested number of elements is zero, but this is not a real use case for it anyway. But the best part is yet to come. Because of the way Python implicitly packs & unpacks tuples, you can use it *without* calling tuple(): >>> a,b = iunpack((1,2,3,4), 1) >>> a,b (1, (2, 3, 4)) >>> a,b,c = iunpack((1,2,3,4), 2) >>> a,b,c (1, 2, (3, 4)) >>> a,b,c,d,e = iunpack((1,2,3,4), 4) >>> a,b,c,d,e (1, 2, 3, 4, ()) >>> a,b,c,d,e,f,g = iunpack((1,2,3,4), 6) >>> a,b,c,d,e,f,g (1, 2, 3, 4, None, None, ()) The only catch is that, if you have only one parameter, then all you will get is the generator itself. >>> a = iunpack((1,2,3,4), 1) >>> a <generator object at 0x4071bfac> But that's a corner case, and not the intended use anyway. Besides that, there is an issue regarding the 'times' parameter; whether it should return 'times' items plus the tail part, or 'times-1' items and the tail part. I think that it's fine the way it is. -- Carlos Ribeiro Consultoria em Projetos blog: http://rascunhosrotos.blogspot.com blog: http://pythonnotes.blogspot.com mail: carribeiro@gmail.com mail: carribeiro@yahoo.com From sxanth at ceid.upatras.gr Thu Nov 18 05:49:31 2004 From: sxanth at ceid.upatras.gr (Stelios Xanthakis) Date: Thu Nov 18 05:49:39 2004 Subject: [Python-Dev] __pycode__ extension In-Reply-To: <5.1.1.6.0.20041117105731.020e4230@mail.telecommunity.com> References: <419B2C76.1000709@iinet.net.au> <419A4C1A.5060502@ceid.upatras.gr> <419B2C76.1000709@iinet.net.au> <5.1.1.6.0.20041117105731.020e4230@mail.telecommunity.com> Message-ID: <Pine.GSO.4.61.0411180639180.28978@diogenis.ceid.upatras.gr> On Wed, 17 Nov 2004, Phillip J. Eby wrote: > >> The __pycode__ attribute is not marshaled. >> So for 'import'ed code, __pycode__=None >> > > What good is that? Likewise, I don't see the point of having this only > enabled when a command-line option is given. We can marshal it if we want. But I'm not convinced we should. Having used this system, 'import' is a good barrier to say "I'm not interested for the __pycode__ of these". For example in the personal shell/library were I collect nice recipies from usenet (premuttating letters and fairy tales), I want to edit the recipe but when I import httplib, I'm not interested in editing standard modules. Afterall, we can execfile() in locals() to bypass import and load a module with pycode. > > If the idea is just to allow saving code from interactive mode, why not just > modify the interactive mode to do this? > I think that is easilly doable by the way tok_nextc() is implemented (in 2.3.4 at least). We can avoid this if we have sourcefile/line. btw, don't forget the exec case: exec """def spam(): x = 1""" in globals(), globals() print spam.__pycode__ So having __pycode__==None can mean that - either there is source available, and use inspect - no source at all because we have pyc only Another issue though is that if we implement this we could modify inspect and drop file/lineno/linetab and go read from __pycode__. From python at rcn.com Thu Nov 18 06:59:12 2004 From: python at rcn.com (Raymond Hettinger) Date: Thu Nov 18 07:01:38 2004 Subject: [Python-Dev] Deprecate PyRange_New() In-Reply-To: <1f7befae04111120514932c693@mail.gmail.com> Message-ID: <000801c4cd33$bafbb220$e841fea9@oemcomputer> > [Raymond Hettinger] > > A while back, Tim added a comment that PyRange_New() should be > > deprecated in-part because it is filled with problems and is not used > > anywhere in the core. [Tim] > And because it's undocumented. The only place it's even mentioned is > in the NEWS for 2.3a1: > > - PyRange_New() now raises ValueError if the fourth argument is not 1. > This is part of the removal of deprecated features of the xrange > object. [Raymond] > > Doesn't anyone mind if I mark it in the C-API docs as deprecated so it > > can be phased out later? [Tim] > +1 > > I don't object to *documenting* a PyRange_New() API function, BTW, but > the current implementation is too broken to fix with less than heroic > effort. If PyRange_New() needs to exist in the C API, then it should > have the same (lo, hi, step) signature as the builtin range(), for > which we have correct but non-obvious error-checking C code. The > funky (start, len, step, reps) signature of the current PyRange_New() > is much more difficult to check for errors (e.g., detecting C long > multiplication overflow in portable C is a sub-problem), and the > current signature is too surprising regardless (because it's so > different from the builtin range()). Given that it is undocumented, broken, and recent, would it be over the top to just remove it from Py2.4? Raymond From pje at telecommunity.com Thu Nov 18 07:15:42 2004 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu Nov 18 07:14:48 2004 Subject: [Python-Dev] __pycode__ extension In-Reply-To: <Pine.GSO.4.61.0411180639180.28978@diogenis.ceid.upatras.gr > References: <5.1.1.6.0.20041117105731.020e4230@mail.telecommunity.com> <419B2C76.1000709@iinet.net.au> <419A4C1A.5060502@ceid.upatras.gr> <419B2C76.1000709@iinet.net.au> <5.1.1.6.0.20041117105731.020e4230@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20041118010844.03806cb0@mail.telecommunity.com> At 06:49 AM 11/18/04 +0200, Stelios Xanthakis wrote: >On Wed, 17 Nov 2004, Phillip J. Eby wrote: >>What good is that? Likewise, I don't see the point of having this only >>enabled when a command-line option is given. > >We can marshal it if we want. >But I'm not convinced we should. Apparently, we have different use cases for the source code. In my use cases, I have little use for the interactive mode or exec. For example, in the exec case, I would already have the source code, so why would I need it again? I think it's important to clarify what this is *for*, and then address whatever that use case is directly. So far, lots of people are projecting their own use cases onto it, and therefore we all disagree on what the best way to implement it is. > Having used this system, >'import' is a good barrier to say "I'm not interested for >the __pycode__ of these". Having used it for whatever purposes *you're* using it for. Evidently you want to use it for something very different from what I would want it for (source code transforms, such as AOP "advice" injection). From ncoghlan at iinet.net.au Thu Nov 18 11:44:07 2004 From: ncoghlan at iinet.net.au (Nick Coghlan) Date: Thu Nov 18 11:44:12 2004 Subject: [Python-Dev] __pycode__ extension In-Reply-To: <5.1.1.6.0.20041118010844.03806cb0@mail.telecommunity.com> References: <5.1.1.6.0.20041117105731.020e4230@mail.telecommunity.com> <419B2C76.1000709@iinet.net.au> <419A4C1A.5060502@ceid.upatras.gr> <419B2C76.1000709@iinet.net.au> <5.1.1.6.0.20041117105731.020e4230@mail.telecommunity.com> <5.1.1.6.0.20041118010844.03806cb0@mail.telecommunity.com> Message-ID: <419C7CF7.1040808@iinet.net.au> Phillip J. Eby wrote: > At 06:49 AM 11/18/04 +0200, Stelios Xanthakis wrote: > > I think it's important to clarify what this is *for*, and then address > whatever that use case is directly. So far, lots of people are > projecting their own use cases onto it, and therefore we all disagree on > what the best way to implement it is. True - hopefully a PEP can flush out most of the use cases. >> Having used this system, >> 'import' is a good barrier to say "I'm not interested for >> the __pycode__ of these". > > > Having used it for whatever purposes *you're* using it for. Evidently > you want to use it for something very different from what I would want > it for (source code transforms, such as AOP "advice" injection). The interface I envisioned with my last message was something like: exec compile(source_str, source_file, 'exec', compile.STORE_SOURCE) However, I've now discovered that the flags argument to compile is currently used only for controlling which __future__ statements are in effect. So this would be a significant change, as it would using a portion of the __future__ machinery to control something that is never intended to become default behaviour. With this approach (or something similar, like a separate 'store=compile.SOURCE' argument to compile instead of piggybacking the future machinery), normal code remains unaffected, but an application can enable the functionality when it wants it by compiling the source code directly with the option enabled. It may be reasonable to alter the interactive shell to enable the option for code entered at the interactive prompt. This way, code compiled normally does not get its source stored in memory or the pyc files. Code which needs the source to operate correctly can be compiled with the option, and the relevant parts of the source would then be stored With the ast-branch in, similar functionality could be provided for 'compile.STORE_AST' or 'store=compile.AST' (OR the flags to store both source and AST). At-least-we-don't-have-to-worry-about-getting-enough-material-for-a-PEP'ly, Nick. -- Nick Coghlan | Brisbane, Australia Email: ncoghlan@email.com | Mobile: +61 409 573 268 From ncoghlan at iinet.net.au Thu Nov 18 12:07:34 2004 From: ncoghlan at iinet.net.au (Nick Coghlan) Date: Thu Nov 18 12:07:40 2004 Subject: [Python-Dev] syntactic shortcut - unpack to variably sized list In-Reply-To: <864d3709041117184370f63c1b@mail.gmail.com> References: <000c01c4c838$77bb3da0$0d00a8c0@MAINFRAME> <864d3709041117184370f63c1b@mail.gmail.com> Message-ID: <419C8276.5030809@iinet.net.au> Carlos Ribeiro wrote: > On Thu, 11 Nov 2004 22:50:21 +0100, Johan Hahn <johahn@home.se> wrote: > >>Hi >> >>As far as I can tell from the archive, this has not been discussed before. >>This is the second time in less than a week that I have stumbled over the rather >>clumsy syntax of extracting some elements of a sequence and at the same time >>remove those from the sequence: >> >>>>>L = 'a b 1 2 3'.split(' ') >>>>>a,b,L = L[0], L[1], L[2:] > > > I am really late on this thread, but anyway, I've come up with another > approach to solve the problem using iterators. It uses iterator that > is guaranteed to always return a fixed number of elements, regardless > of the size of the sequence; when it finishes, it returns the tail of > the sequence as the last argument. This is a simple-minded proof of > concept, and it's surely highly optimizable in at least a hundred > different ways :-) So the original example becomes: a, b, L = itertools.iunpack(L, 2) (and the 2 is an element count, not an index, so, as you say, there's no need to subtract 1) That's certainly quite tidy, but it has the downside that it still copies the entire list as happens in the OP's code (the copying is hidden, but it still happens - the original list isn't destroyed until the assignment of the third value returned by the iterator). It's also likely to require a trip to the docs to find out how iunpack works, and it doesn't fare well in the 'discovery' category (that is, the solution to what should be a fairly basic list operation is tucked away in itertools) If list.pop gets updated to handle slice objects, then it can modify the list in place, and avoid any copying of list elements. "a, b = L.pop(slice(2)" should be able to give even better performance than "a = L.pop(0); b = L.pop(0)" (which is, I believe, the only current approach that avoids copying the entire list). And the list operation stays where it belongs - in a method of list. Cheers, Nick. -- Nick Coghlan | Brisbane, Australia Email: ncoghlan@email.com | Mobile: +61 409 573 268 From ncoghlan at iinet.net.au Thu Nov 18 12:21:11 2004 From: ncoghlan at iinet.net.au (Nick Coghlan) Date: Thu Nov 18 12:21:16 2004 Subject: [Python-Dev] Re: subprocess and EINTR errnos In-Reply-To: <Pine.GSO.4.51L2.0411162121580.11957@koeberg.lysator.liu.se> References: <20041111034803.GA16175@isc.upenn.edu> <Pine.GSO.4.51L2.0411162121580.11957@koeberg.lysator.liu.se> Message-ID: <419C85A7.70304@iinet.net.au> Peter Astrand wrote: > I've made a patch (attached) to subprocess.py (and test_subprocess.py) > that should guard against EINTR, but I haven't committed it yet. It's > quite large. This is probably outside the bounds of the "conservative" checkins Anthony has requested prior to the release of 2.4. Given also that it may affect more than just subprocess.py, this looks like a 2.5 problem. Cheers, Nick. -- Nick Coghlan | Brisbane, Australia Email: ncoghlan@email.com | Mobile: +61 409 573 268 From carribeiro at gmail.com Thu Nov 18 12:56:49 2004 From: carribeiro at gmail.com (Carlos Ribeiro) Date: Thu Nov 18 12:57:03 2004 Subject: [Python-Dev] syntactic shortcut - unpack to variably sized list In-Reply-To: <419C8276.5030809@iinet.net.au> References: <000c01c4c838$77bb3da0$0d00a8c0@MAINFRAME> <864d3709041117184370f63c1b@mail.gmail.com> <419C8276.5030809@iinet.net.au> Message-ID: <864d370904111803561c650864@mail.gmail.com> On Thu, 18 Nov 2004 21:07:34 +1000, Nick Coghlan <ncoghlan@iinet.net.au> wrote: > Carlos Ribeiro wrote: > > > > On Thu, 11 Nov 2004 22:50:21 +0100, Johan Hahn <johahn@home.se> wrote: > > > >>Hi > >> > >>As far as I can tell from the archive, this has not been discussed before. > >>This is the second time in less than a week that I have stumbled over the rather > >>clumsy syntax of extracting some elements of a sequence and at the same time > >>remove those from the sequence: > >> > >>>>>L = 'a b 1 2 3'.split(' ') > >>>>>a,b,L = L[0], L[1], L[2:] > > > > > > I am really late on this thread, but anyway, I've come up with another > > approach to solve the problem using iterators. It uses iterator that > > is guaranteed to always return a fixed number of elements, regardless > > of the size of the sequence; when it finishes, it returns the tail of > > the sequence as the last argument. This is a simple-minded proof of > > concept, and it's surely highly optimizable in at least a hundred > > different ways :-) > > So the original example becomes: > a, b, L = itertools.iunpack(L, 2) > > (and the 2 is an element count, not an index, so, as you say, there's no need to > subtract 1) > > That's certainly quite tidy, but it has the downside that it still copies the > entire list as happens in the OP's code (the copying is hidden, but it still > happens - the original list isn't destroyed until the assignment of the third > value returned by the iterator). > > It's also likely to require a trip to the docs to find out how iunpack works, > and it doesn't fare well in the 'discovery' category (that is, the solution to > what should be a fairly basic list operation is tucked away in itertools) > > If list.pop gets updated to handle slice objects, then it can modify the list in > place, and avoid any copying of list elements. "a, b = L.pop(slice(2)" should be > able to give even better performance than "a = L.pop(0); b = L.pop(0)" (which > is, I believe, the only current approach that avoids copying the entire list). > And the list operation stays where it belongs - in a method of list. list.pop doesn't solve the case for when the data is stored in a tuple (which is immutable). Also, a C implementation (of the type that would be done for itertools inclusion) would certainly solve some of the performance issues. As far as the documentation being tucked away into itertools, I don't see that as a problem; in my opinion, itertools already holds a good deal of utility functions which deserve better study by novices (on a near par with builtins). And finally, I believe that the name iunpack is already quite obvious on what it means. BTW, just because map, zip, enumerate, filter & etc are in builtins doesn't mean that they are inherently more useful, or pythonic, than the functions in itertools. It's just a matter that itertools is a relatively late addition to the pack; I hope it becomes more proeminently adopted as people get used to them. -- Carlos Ribeiro Consultoria em Projetos blog: http://rascunhosrotos.blogspot.com blog: http://pythonnotes.blogspot.com mail: carribeiro@gmail.com mail: carribeiro@yahoo.com From barry at python.org Thu Nov 18 15:06:28 2004 From: barry at python.org (Barry Warsaw) Date: Thu Nov 18 15:06:32 2004 Subject: [Python-Dev] Deprecate PyRange_New() In-Reply-To: <000801c4cd33$bafbb220$e841fea9@oemcomputer> References: <000801c4cd33$bafbb220$e841fea9@oemcomputer> Message-ID: <1100786788.9273.194.camel@presto.wooz.org> On Thu, 2004-11-18 at 00:59, Raymond Hettinger wrote: > Given that it is undocumented, broken, and recent, would it be over the > top to just remove it from Py2.4? Given that 2.4 is (or nearly is) a release candidate, do you even need to ask that? :) -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 307 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-dev/attachments/20041118/76603f9e/attachment.pgp From sxanth at ceid.upatras.gr Thu Nov 18 15:14:04 2004 From: sxanth at ceid.upatras.gr (Stelios Xanthakis) Date: Thu Nov 18 15:18:24 2004 Subject: [Python-Dev] __pycode__ extension In-Reply-To: <5.1.1.6.0.20041118010844.03806cb0@mail.telecommunity.com> References: <5.1.1.6.0.20041117105731.020e4230@mail.telecommunity.com> <419B2C76.1000709@iinet.net.au> <419A4C1A.5060502@ceid.upatras.gr> <419B2C76.1000709@iinet.net.au> <5.1.1.6.0.20041117105731.020e4230@mail.telecommunity.com> <5.1.1.6.0.20041118010844.03806cb0@mail.telecommunity.com> Message-ID: <Pine.GSO.4.61.0411180901380.3835@zenon.ceid.upatras.gr> On Thu, 18 Nov 2004, Phillip J. Eby wrote: > > I think it's important to clarify what this is *for*, and then address > whatever that use case is directly. > Ok. Let's see it from the "programming languages" prespective. In my opinion with the addition of __pycode__ (or __source__ from now on), we have a good emulation of "data is code and code is data", just like machine code. In python we already have "data is code". X='''def f(): pass''' exec X But because of the conversion to bytecode we do not have the "code is data" part (unless we consider code object data). Why is this useful? I think it's one of the benefits of interpreted vs compiled languages.. The real thing would be to attach __source__ to code objects, but I think __source__ in functions and classes only is more realistic. Anyway this is getting too theoretical and we can discuss it forever on 'what is and what isn't code and data'.. As I said before: This *can* already be done in python as it is by writing some kind of framework/IDE (Zope, Plone, IDLE, ipython,whatever). With __source__ though, python becomes a framework, we achieve this with the minimum overhead and we are sure it will work with all cases (interactive/exec) and all future extensions. On the issue of saving __source__ to .pyc files, I'm 50-50. I'd vote for -0. Also I'd vote +1 for a command line option which if specified __source__ will be generated for *all* functions/classes (not only interactively defined). Now I will go see about a PEP so that we can stop referring to it as "it" and say "PEP xxx":) Stelios From pje at telecommunity.com Thu Nov 18 16:33:25 2004 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu Nov 18 16:32:33 2004 Subject: [Python-Dev] __pycode__ extension In-Reply-To: <Pine.GSO.4.61.0411180901380.3835@zenon.ceid.upatras.gr> References: <5.1.1.6.0.20041118010844.03806cb0@mail.telecommunity.com> <5.1.1.6.0.20041117105731.020e4230@mail.telecommunity.com> <419B2C76.1000709@iinet.net.au> <419A4C1A.5060502@ceid.upatras.gr> <419B2C76.1000709@iinet.net.au> <5.1.1.6.0.20041117105731.020e4230@mail.telecommunity.com> <5.1.1.6.0.20041118010844.03806cb0@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20041118102638.03d79b50@mail.telecommunity.com> At 04:14 PM 11/18/04 +0200, Stelios Xanthakis wrote: >On Thu, 18 Nov 2004, Phillip J. Eby wrote: >> >>I think it's important to clarify what this is *for*, and then address >>whatever that use case is directly. > >Ok. Let's see it from the "programming languages" prespective. >In my opinion with the addition of __pycode__ (or __source__ from >now on), we have a good emulation of "data is code and code is data", >just like machine code. [snip lots of explanation] Your explanation still hasn't answered the question of what you plan to *do* with __source__. Absent that, there are no criteria for making design decisions regarding the implementation. >As I said before: This *can* already be done in python as it is by >writing some kind of framework/IDE (Zope, Plone, IDLE, ipython,whatever). In the absence of use cases, this is an argument for *not* making a change to the Python core. >On the issue of saving __source__ to .pyc files, I'm 50-50. >I'd vote for -0. > >Also I'd vote +1 for a command line option which if specified >__source__ will be generated for *all* functions/classes >(not only interactively defined). And I'd vote differently on both these matters. Please state what your use cases are, so that solutions can be evaluated on the basis of what use cases they satisfy, not merely votes without any context. >Now I will go see about a PEP so that we can stop referring >to it as "it" and say "PEP xxx":) Please take especial care to include in the Motivation section what *specific* use cases this is intended to address. So far the *only* use case you have presented is saving functions that were entered interactively, but this can be accomplished in far less controversial ways. From pedronis at bluewin.ch Thu Nov 18 16:52:35 2004 From: pedronis at bluewin.ch (Samuele Pedroni) Date: Thu Nov 18 16:49:31 2004 Subject: [Python-Dev] __pycode__ extension In-Reply-To: <5.1.1.6.0.20041118102638.03d79b50@mail.telecommunity.com> References: <5.1.1.6.0.20041118010844.03806cb0@mail.telecommunity.com> <5.1.1.6.0.20041117105731.020e4230@mail.telecommunity.com> <419B2C76.1000709@iinet.net.au> <419A4C1A.5060502@ceid.upatras.gr> <419B2C76.1000709@iinet.net.au> <5.1.1.6.0.20041117105731.020e4230@mail.telecommunity.com> <5.1.1.6.0.20041118010844.03806cb0@mail.telecommunity.com> <5.1.1.6.0.20041118102638.03d79b50@mail.telecommunity.com> Message-ID: <419CC543.1020405@bluewin.ch> Phillip J. Eby wrote: > > And I'd vote differently on both these matters. Please state what your > use cases are, so that solutions can be evaluated on the basis of what > use cases they satisfy, not merely votes without any context. > I don't get why your are particularly opposed to attaching source for exec-ed/eval-ed code, I think that expanding the cases where inspect.getsource just work directly is valuable. And doing that seem a natural way to achieve this. OTOH I personally fully appreciate why you may want to have it work even if just pycs are around. Stelios Xanthakis wrote: > Having used this system, > 'import' is a good barrier to say "I'm not interested for > the __pycode__ of these". but for example having access to co_filename and or equilavent (something would have to be for done for classes about this) is probably also good enough to make that distinction: >>> def f(): pass ... >>> f.func_code.co_filename '<stdin>' >>> From python at rcn.com Thu Nov 18 18:36:27 2004 From: python at rcn.com (Raymond Hettinger) Date: Thu Nov 18 18:39:03 2004 Subject: [Python-Dev] Deprecate PyRange_New() In-Reply-To: <1100786788.9273.194.camel@presto.wooz.org> Message-ID: <001001c4cd95$2249ce60$e841fea9@oemcomputer> > > Given that it is undocumented, broken, and recent, would it be over the > > top to just remove it from Py2.4? > > Given that 2.4 is (or nearly is) a release candidate, do you even need > to ask that? :) Yes ;-) The code is buggy and undocumented. Fixing bugs and adding documentation are appropriate after the beta and before the release candidate. However, in this case, a complete bugfix would also entail a change in the signature. And since the function is unnecessary, removing it entirely is a safer fix. Leaving it in only increases the risk that someone will find it and link to it. OTOH, I don't care enough about this one to push it. Raymond From rwgk at cci.lbl.gov Thu Nov 18 19:02:55 2004 From: rwgk at cci.lbl.gov (Ralf W. Grosse-Kunstleve) Date: Thu Nov 18 19:03:04 2004 Subject: [Python-Dev] proposal+patch: sys.gettickeraccumulation() Message-ID: <200411181802.iAII2tnl079279@boa.lbl.gov> Brett C. wrote: > I have been mulling over this proposal and I think I am finally > settling on +0 as long as it can be shown that it does not hurt > performance at all (yes, it shouldn't but this is the eval loop we are > talking about and that thing is a touchy beast). Is there a standard benchmark I could use to demonstrate the impact of the two new additions on the runtime? Thanks! Ralf From pje at telecommunity.com Thu Nov 18 19:06:34 2004 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu Nov 18 19:05:45 2004 Subject: [Python-Dev] __pycode__ extension In-Reply-To: <419CC543.1020405@bluewin.ch> References: <5.1.1.6.0.20041118102638.03d79b50@mail.telecommunity.com> <5.1.1.6.0.20041118010844.03806cb0@mail.telecommunity.com> <5.1.1.6.0.20041117105731.020e4230@mail.telecommunity.com> <419B2C76.1000709@iinet.net.au> <419A4C1A.5060502@ceid.upatras.gr> <419B2C76.1000709@iinet.net.au> <5.1.1.6.0.20041117105731.020e4230@mail.telecommunity.com> <5.1.1.6.0.20041118010844.03806cb0@mail.telecommunity.com> <5.1.1.6.0.20041118102638.03d79b50@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20041118125507.031c8c60@mail.telecommunity.com> At 04:52 PM 11/18/04 +0100, Samuele Pedroni wrote: >Phillip J. Eby wrote: >>And I'd vote differently on both these matters. Please state what your >>use cases are, so that solutions can be evaluated on the basis of what >>use cases they satisfy, not merely votes without any context. > >I don't get why your are particularly opposed to attaching source for >exec-ed/eval-ed code, I'm not against it per se. I'm against adding a feature that crudely satisfies certain use cases, but has to be worked around by people who have other use cases. In some cases, a feature can become an "attractive nuisance" that people will use because it's there most of the time, but it then becomes problematic when it's not available. This is especially problematic when the presence or absence of the feature is on a global basis, potentially breaking the composability of Python modules. (I.e., module A relies on the feature's presence, but module B requires its absence.) >I think that expanding the cases where inspect.getsource just work >directly is valuable. Agreed. > And doing that seem a natural way to achieve this. Also agreed - but my concerns are with the larger consequences of having 'getsource' *seem* to work everywhere. >OTOH I personally fully appreciate why you may want to have it work >even if just pycs are around. For me, a half-working __source__ feature is almost worse than none at all, since the former could lead to the illusion of making progress towards code reflection. This is why I'd like it made clearer what problems this is and is not meant to solve, so that programmers who are tempted to use the feature will be clearer as to what is and isn't a good idea to do with it. From pedronis at bluewin.ch Thu Nov 18 19:34:51 2004 From: pedronis at bluewin.ch (Samuele Pedroni) Date: Thu Nov 18 19:31:24 2004 Subject: [Python-Dev] __pycode__ extension In-Reply-To: <5.1.1.6.0.20041118125507.031c8c60@mail.telecommunity.com> References: <5.1.1.6.0.20041118102638.03d79b50@mail.telecommunity.com> <5.1.1.6.0.20041118010844.03806cb0@mail.telecommunity.com> <5.1.1.6.0.20041117105731.020e4230@mail.telecommunity.com> <419B2C76.1000709@iinet.net.au> <419A4C1A.5060502@ceid.upatras.gr> <419B2C76.1000709@iinet.net.au> <5.1.1.6.0.20041117105731.020e4230@mail.telecommunity.com> <5.1.1.6.0.20041118010844.03806cb0@mail.telecommunity.com> <5.1.1.6.0.20041118102638.03d79b50@mail.telecommunity.com> <5.1.1.6.0.20041118125507.031c8c60@mail.telecommunity.com> Message-ID: <419CEB4B.4040306@bluewin.ch> Phillip J. Eby wrote: > >> OTOH I personally fully appreciate why you may want to have it work >> even if just pycs are around. > > > For me, a half-working __source__ feature is almost worse than none at > all, since the former could lead to the illusion of making progress > towards code reflection. This is why I'd like it made clearer what > problems this is and is not meant to solve, so that programmers who are > tempted to use the feature will be clearer as to what is and isn't a > good idea to do with it. > well, personally I think an inspect.getsource that is working and correct as much as possible is a worthy goal. Or a new getast. At this point I see these possible levels of implementation/functionality: 0) disabled 1) attach source only to exec-ed/eval-ed code (with this level inspect.getsource would still do the wrong thing in the case for example of editing) 2) try to always attach source code, don't store it in pyc files, for correct behavior a likely approach is to read pyc and py files in parallel 3) always attach source code, store source in some form in the pyc files I repeat that other means can be devised than flavor 1 of implementation, to distinguish exec-ed/eval-ed/interactive code from code from .py files. Because of people with memory/disk usage considerations and people preferring no-source shipping we may have to implement to some extent all of 0,2,3. Unless e.g. storing ASTs instead of source will be deemed enough safe, or the fact that in 3 the source is not really just open-text-editor away (I doubt that). From rsenra at acm.org Thu Nov 18 23:16:06 2004 From: rsenra at acm.org (Rodrigo Dias Arruda Senra) Date: Thu Nov 18 23:16:25 2004 Subject: [Python-Dev] __pycode__ extension In-Reply-To: <419BE278.7030402@bluewin.ch> References: <419B2C76.1000709@iinet.net.au> <419A4C1A.5060502@ceid.upatras.gr> <419B2C76.1000709@iinet.net.au> <5.1.1.6.0.20041117105731.020e4230@mail.telecommunity.com> <419BA86E.70309@bluewin.ch> <1100729964.20177.29.camel@geddy.wooz.org> <419BE278.7030402@bluewin.ch> Message-ID: <20041118201606.69277207@Goku> [ Samuele Pedroni <pedronis@bluewin.ch> ] ----------------------------------------------- | - people will still want a way to ship without shipping the source, or | such that reconstructing the source is not too easy, with varying | degrees of what is considered enough "secure" Instead of adding source code to .pyc, why not just add references to the original .py file inside the runtime PyObjects (iff cmdline directive is present). If the .py module is not present, then the absence of source *should* be respected. cheers, Rod Senra -- ,_ | ) Rodrigo Senra <rsenra |at| acm.org> |(______ ----------------------------------------------- _( (|__|] GPr Sistemas http://www.gpr.com.br _ | (|___|] IC - Unicamp http://www.ic.unicamp.br/~921234 ___ (|__|] L___(|_|] ----------------------------------------------- From doko at cs.tu-berlin.de Fri Nov 19 01:21:42 2004 From: doko at cs.tu-berlin.de (Matthias Klose) Date: Fri Nov 19 01:21:52 2004 Subject: [Python-Dev] python 2.3.5 release? Message-ID: <16797.15510.883083.658735@gargle.gargle.HOWL> Python 2.3.4 was released on 2004-05-27. According to the bi-annual release schedule that would assume another release from the 2.3 branch this year. Are people still interested in a formal 2.3.5 release? At least the 2.3 branch does have a couple of patches worth making a release. Is there anything that needs to be added before a 2.3.5 release is scheduled, or should we wait for the 2.4 release to identify patches that need to be backported to a potential 2.3.5 release? Thanks for any comment, Matthias From barry at python.org Fri Nov 19 01:27:59 2004 From: barry at python.org (Barry Warsaw) Date: Fri Nov 19 01:28:03 2004 Subject: [Python-Dev] python 2.3.5 release? In-Reply-To: <16797.15510.883083.658735@gargle.gargle.HOWL> References: <16797.15510.883083.658735@gargle.gargle.HOWL> Message-ID: <1100824079.8787.89.camel@geddy.wooz.org> On Thu, 2004-11-18 at 19:21, Matthias Klose wrote: > Python 2.3.4 was released on 2004-05-27. According to the bi-annual > release schedule that would assume another release from the 2.3 branch > this year. Are people still interested in a formal 2.3.5 release? Yes, and I think Anthony is planning to do one a few weeks after 2.4 final. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 307 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-dev/attachments/20041118/e7a31ef2/attachment.pgp From tim.peters at gmail.com Fri Nov 19 01:30:35 2004 From: tim.peters at gmail.com (Tim Peters) Date: Fri Nov 19 01:30:37 2004 Subject: [Python-Dev] python 2.3.5 release? In-Reply-To: <16797.15510.883083.658735@gargle.gargle.HOWL> References: <16797.15510.883083.658735@gargle.gargle.HOWL> Message-ID: <1f7befae0411181630376522a9@mail.gmail.com> Anthony announced his intent to produce a 2.3.5 release, after 2.4 final. Brett announced his intent to send a "one-month warning" about 2.3.5 so people could lobby for patches. Since I haven't seen such a warning yet, 2.3.5 must be at least a month away <wink>. From tim.peters at gmail.com Fri Nov 19 03:08:20 2004 From: tim.peters at gmail.com (Tim Peters) Date: Fri Nov 19 03:08:23 2004 Subject: [Python-Dev] New critical thread bug Message-ID: <1f7befae041118180859aabf4a@mail.gmail.com> This one is a puzzler. See http://www.python.org/sf/1069160 for details. The short course is that the PyThreadState_SetAsyncExc() implementation fell into a common trap, and can cause segfaults under rare conditions (like every other Python thread segfault bug we've ever had). This is easily repaired (although I've got no interest in doing the coding, or even in contriving a test case -- this was an obvious "by eyeball" bug). The puzzle is how to treat this wrt 2.4. Since it's a critical bug, I suppose it "should" force another release candidate. OTOH, this is a C-only API (there's no exposure from Python) that's never used in the core. We could add code to make it segfault every time <wink>, and nothing in the distribution would notice. OTOH, if we broke its intended behavior while fixing the bug, we'd never know that either. "Never used in the core" means never -- the function isn't tested. On the third hand, it's a simple function with an obvious segfault mode that has an obvious fix. I'll leave it to the release manager <wink>. From anthony at interlink.com.au Fri Nov 19 04:57:00 2004 From: anthony at interlink.com.au (Anthony Baxter) Date: Fri Nov 19 05:00:23 2004 Subject: [Python-Dev] python 2.3.5 release? In-Reply-To: <1f7befae0411181630376522a9@mail.gmail.com> References: <16797.15510.883083.658735@gargle.gargle.HOWL> <1f7befae0411181630376522a9@mail.gmail.com> Message-ID: <419D6F0C.3020102@interlink.com.au> Tim Peters wrote: > Anthony announced his intent to produce a 2.3.5 release, after 2.4 > final. Brett announced his intent to send a "one-month warning" about > 2.3.5 so people could lobby for patches. Since I haven't seen such a > warning yet, 2.3.5 must be at least a month away <wink>. At the current time, it looks likely that this will be in early January. Anthony -- Anthony Baxter <anthony@interlink.com.au> It's never too late to have a happy childhood. From bac at OCF.Berkeley.EDU Fri Nov 19 05:01:42 2004 From: bac at OCF.Berkeley.EDU (Brett C.) Date: Fri Nov 19 05:01:47 2004 Subject: [Python-Dev] proposal+patch: sys.gettickeraccumulation() In-Reply-To: <200411181802.iAII2tnl079279@boa.lbl.gov> References: <200411181802.iAII2tnl079279@boa.lbl.gov> Message-ID: <419D7026.6010007@ocf.berkeley.edu> Ralf W. Grosse-Kunstleve wrote: > Brett C. wrote: > >>I have been mulling over this proposal and I think I am finally >>settling on +0 as long as it can be shown that it does not hurt >>performance at all (yes, it shouldn't but this is the eval loop we are >>talking about and that thing is a touchy beast). > > > Is there a standard benchmark I could use to demonstrate the > impact of the two new additions on the runtime? > Thanks! =) Parrotbench and PyBench would be enough for me to sign off on any performance hit. There is also pystone. -Brett From bac at OCF.Berkeley.EDU Fri Nov 19 05:09:31 2004 From: bac at OCF.Berkeley.EDU (Brett C.) Date: Fri Nov 19 05:09:46 2004 Subject: [Python-Dev] python 2.3.5 release? In-Reply-To: <419D6F0C.3020102@interlink.com.au> References: <16797.15510.883083.658735@gargle.gargle.HOWL> <1f7befae0411181630376522a9@mail.gmail.com> <419D6F0C.3020102@interlink.com.au> Message-ID: <419D71FB.3040605@ocf.berkeley.edu> Anthony Baxter wrote: > Tim Peters wrote: > >> Anthony announced his intent to produce a 2.3.5 release, after 2.4 >> final. Brett announced his intent to send a "one-month warning" about >> 2.3.5 so people could lobby for patches. Since I haven't seen such a >> warning yet, 2.3.5 must be at least a month away <wink>. > > > At the current time, it looks likely that this will be in early January. > I will probably make some announcement in early December to remind people to get stuff in, especially with people having Winter vacation and thus either having less time because they are off galavanting around or more because they get to relax in front of the fire coding. Anthony, can you keep me posted if you think it will slip much past early January? -Brett From bac at OCF.Berkeley.EDU Fri Nov 19 05:10:53 2004 From: bac at OCF.Berkeley.EDU (Brett C.) Date: Fri Nov 19 05:11:16 2004 Subject: [Python-Dev] python 2.3.5 release? In-Reply-To: <419D6F0C.3020102@interlink.com.au> References: <16797.15510.883083.658735@gargle.gargle.HOWL> <1f7befae0411181630376522a9@mail.gmail.com> <419D6F0C.3020102@interlink.com.au> Message-ID: <419D724D.2080600@ocf.berkeley.edu> Anthony Baxter wrote: > Tim Peters wrote: > >> Anthony announced his intent to produce a 2.3.5 release, after 2.4 >> final. Brett announced his intent to send a "one-month warning" about >> 2.3.5 so people could lobby for patches. Since I haven't seen such a >> warning yet, 2.3.5 must be at least a month away <wink>. > > > At the current time, it looks likely that this will be in early January. > I will probably make some announcement in early December to remind people to get stuff in, especially with people having Winter vacation and thus either having less time because they are gallivanting around or more because they get to relax in front of the fire coding. Anthony, can you keep me posted if you think it will slip much past early January? -Brett From anthony at interlink.com.au Fri Nov 19 08:16:45 2004 From: anthony at interlink.com.au (Anthony Baxter) Date: Fri Nov 19 08:17:05 2004 Subject: [Python-Dev] RELEASED Python 2.4, release candidate 1 Message-ID: <200411191816.52309.anthony@interlink.com.au> On behalf of the Python development team and the Python community, I'm happy to announce the first release candidate of Python 2.4. Python 2.4c1 is a release candidate - we'd greatly appreciate it if you could download it, kick the tires and let us know of any problems you find, but it is probably not suitable for production usage. At this point, we're particularly interested in build bugs. If you could grab the source archive, build it and then run the test suite, we'd appreciate it. http://www.python.org/2.4 Notable changes in rc1 include a handful of bug fixes, including a thread shutdown race bug. See either the highlights, the What's New in Python 2.4, or the detailed NEWS file -- all available from the Python 2.4 webpage. This is the first release candidate of Python 2.4. Python 2.4 is now close to release - there should be few or no new features, merely bug fixes. Assuming no problems, we're looking at a final release around the end of November (if there are problems, there will be another release candidate). Please log any problems you have with this release in the SourceForge bug tracker (noting that you're using 2.4c1): http://sourceforge.net/bugs/?group_id=5470 Enjoy the new release, Anthony Anthony Baxter anthony@python.org Python Release Manager (on behalf of the entire python-dev team) -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://mail.python.org/pipermail/python-dev/attachments/20041119/9d4db1e6/attachment.pgp From ncoghlan at iinet.net.au Fri Nov 19 11:29:45 2004 From: ncoghlan at iinet.net.au (Nick Coghlan) Date: Fri Nov 19 11:29:49 2004 Subject: [Python-Dev] syntactic shortcut - unpack to variably sized list In-Reply-To: <864d370904111803561c650864@mail.gmail.com> References: <000c01c4c838$77bb3da0$0d00a8c0@MAINFRAME> <864d3709041117184370f63c1b@mail.gmail.com> <419C8276.5030809@iinet.net.au> <864d370904111803561c650864@mail.gmail.com> Message-ID: <419DCB19.5000609@iinet.net.au> Carlos Ribeiro wrote: > list.pop doesn't solve the case for when the data is stored in a > tuple (which is immutable). For immutable objects, you *have* to make a copy, so I don't see any real downside to just using: a, b, T = T[0], T[1], T[2:] While I think iunpack is kind of neat, I don't think it solves anything which is currently a major problem, as it is really just a different way of spelling the above slicing. The major portion (sans some index checking) of iunpack(T, 2) can be written on one line: a, b, T = (T[i] for i in (range(2) + [slice(2, None)])) When the number of elements to be unpacked is known at compile time (as it has to be to use tuple unpacking assignment), there seems little benefit in moving things inside a generator instead of spelling them out as a tuple of slices. However, the OP's original question related to a list, where the goal was to both read the first couple of elements *and* remove them from the list (i.e. 'pop'ing them). This is similar to the motivating use cases which came up the last time the "a, b, *c = L" idea was raised. At the moment, the main choices are to use the syntax above (and make an unnecessary copy), make multiple calls to list.pop (with associated function overhead), or call del directly after reading the slice you want (i.e. doing a list.pop of a slice in Python code). According to the current docs, L.pop(i) is defined as being equivalent to: x = L[i]; del L[i]; return x And Armin pointed out that setting "i = slice(<whatever>)" actually causes that stated equivalence to break: >>> L = [1, 2, 3, 4] >>> i = slice(2) >>> L.pop(i) Traceback (most recent call last): File "<stdin>", line 1, in ? TypeError: an integer is required >>> L[i] [1, 2] >>> del L[i] >>> L [3, 4] >>> So, we could either fix the docs to explicitly exclude slices from list.pop, or just fix the method so slices work as the argument. The latter is a little more work, but it does provides a nice way of spelling "read and remove" for a range of elements. Which is why I'm hoping one of the proponents of "a, b, *c = L" will be willing to provide an implementation. Cheers, Nick. -- Nick Coghlan | Brisbane, Australia Email: ncoghlan@email.com | Mobile: +61 409 573 268 From anthony at interlink.com.au Fri Nov 19 12:03:52 2004 From: anthony at interlink.com.au (Anthony Baxter) Date: Fri Nov 19 12:22:26 2004 Subject: [Python-Dev] TRUNK UNFROZEN Message-ID: <200411192203.52490.anthony@interlink.com.au> Ok, it's been long enough without any screamingly obvious problems, so I'm declaring the trunk unfrozen. Thanks to Tim's latest discovery, it looks like we might need an rc2 - I might try to aim for this in a week's time (assuming that's ok with Fred and Martin) and try to keep things moving along to 2.4 final. After which, I'm going to go out and get very drunk <wink> Anthony From ncoghlan at iinet.net.au Fri Nov 19 16:03:42 2004 From: ncoghlan at iinet.net.au (Nick Coghlan) Date: Fri Nov 19 16:03:49 2004 Subject: [Python-Dev] Current CVS, Cygwin and "make test" Message-ID: <419E0B4E.7040109@iinet.net.au> This was meant to be a "Yup, she's all good" exercise before 2.4 final goes out the door. Not quite so simple, unfortunately. . . (FWIW, I doubt this is a Python problem, but I'd like some of the other folks here who use Cygwin to have a look at the failure symptoms) My Python build (built & running under Cygwin) seems to be having trouble with any tests involving threads (test_asynchat, test_thread, test_threaded_import, test_socket_ssl for starters). When run using "./python -E Lib/test/regrtest.py -unetwork,decimal -r", several tests execute normally, then the test execution hangs. From what I can tell, the hang generally occurs on the first test that makes use of threading. When run on their own, the affected tests run for a short while, then terminate abruptly (i.e. regrtest.py doesn't print its result summary, and the output from the tests themselves shows they haven't run to completion). I suspect (hope?) that the problem is actually in my Cygwin setup rather than in Python, but I'm trying to figure out how to go about proving that. I did a 'make clean' and rebuild to see if stale files were the problem, then a 'make install' to see if the problem was accidentally picking the old python2.3 DLL's instead of the python2.4 DLL's, but neither of those appeared to make any difference. I only got the regression tests working *at all* today, after doing 'find . -name "\.dll" | rebaseall -T -' from my python/dist/src directory to get rid of the Cygwin remap errors when loading DLL's. I figured I'd ask the folks here for ideas (is Jason still around?), before trying to ask the question from the Cygwin end. Cheers, Nick. -- Nick Coghlan | Brisbane, Australia Email: ncoghlan@email.com | Mobile: +61 409 573 268 From carribeiro at gmail.com Fri Nov 19 16:24:48 2004 From: carribeiro at gmail.com (Carlos Ribeiro) Date: Fri Nov 19 16:24:51 2004 Subject: [Python-Dev] syntactic shortcut - unpack to variably sized list In-Reply-To: <419DCB19.5000609@iinet.net.au> References: <000c01c4c838$77bb3da0$0d00a8c0@MAINFRAME> <864d3709041117184370f63c1b@mail.gmail.com> <419C8276.5030809@iinet.net.au> <864d370904111803561c650864@mail.gmail.com> <419DCB19.5000609@iinet.net.au> Message-ID: <864d37090411190724440dfe38@mail.gmail.com> On Fri, 19 Nov 2004 20:29:45 +1000, Nick Coghlan <ncoghlan@iinet.net.au> wrote: > Carlos Ribeiro wrote: > > list.pop doesn't solve the case for when the data is stored in a > > tuple (which is immutable). > > For immutable objects, you *have* to make a copy, so I don't see any real > downside to just using: > > a, b, T = T[0], T[1], T[2:] (First of all, let me say that I am not at all against making list.pop accept slices. It's nice that it seems to be agreed that this is a worthy addition. And I don't intend to keep pushing for iunpack(); but I feel that is important to make a few clarifications.) If I want to do: a,b,X = T[0], T[1], T[2:] (?) The lesson here is that it can't be assumed for unpacking purposes that the programmer wanted to modify the original list. Of course, if he does want it, then list.pop just comes nice. (also, list.pop(slice) can be used to remove elements from anywhere on the list, which is a big plus) > While I think iunpack is kind of neat, I don't think it solves anything which is > currently a major problem, as it is really just a different way of spelling the > above slicing. The major portion (sans some index checking) of iunpack(T, 2) can > be written on one line: > > a, b, T = (T[i] for i in (range(2) + [slice(2, None)])) I would rather prefer not to have to use this idiom -- it's neither obvious nor convenient. Either list.pop or unpack would be cleaner for this purpose. > When the number of elements to be unpacked is known at compile time (as it has > to be to use tuple unpacking assignment), there seems little benefit in moving > things inside a generator instead of spelling them out as a tuple of slices. For more than a few arguments, it seems to be silly to require the user to write it as: a,b,c,d,e = t[0],t[1],t[2],t[3],t[4:] [lots of remarks about list.pop snipped] I'm not against list.pop; it's just that iunpack provides a different approach to the problem. It's fairly generic: it works for both mutable and immutable lists. The implementation provided is a proof of concept. The fact that it does not modify the original arguments *is* a design feature (althought not what the OP wanted for his particular case). After all, as shown in the example above, conventional tuple unpacking on assignment doesn't change the right-side arguments either. One posible improvement for iunpack() is to accept any iterable as an argument. On return, instead of a tuple to represent the remaining elements, it would return the generator itself. This would allow for code that consume the generator in "chunks" to be written rather simply: a,b,c,g = iunpack(3, g) In this case, to solve the OP problem, instead to have all the values to be processed stored into a list, he could rely on a generator to produce values on the fly. Depending on the scenario this would come handy (less memory consumption, more responsiveness). -- Carlos Ribeiro Consultoria em Projetos blog: http://rascunhosrotos.blogspot.com blog: http://pythonnotes.blogspot.com mail: carribeiro@gmail.com mail: carribeiro@yahoo.com From tim.peters at gmail.com Fri Nov 19 18:37:13 2004 From: tim.peters at gmail.com (Tim Peters) Date: Fri Nov 19 18:37:15 2004 Subject: [Python-Dev] TRUNK UNFROZEN In-Reply-To: <200411192203.52490.anthony@interlink.com.au> References: <200411192203.52490.anthony@interlink.com.au> Message-ID: <1f7befae04111909376b170b56@mail.gmail.com> [Anthony Baxter] ... > Thanks to Tim's latest discovery, it looks like we might need an rc2 - > I might try to aim for this in a week's time (assuming that's ok with > Fred and Martin) and try to keep things moving along to 2.4 final. > After which, I'm going to go out and get very drunk <wink> FWIW, I wouldn't bother with an rc2 just for the PyThreadState_SetAsyncExc segfault bug. It's never been reported in real life, it can't occur unless a 3rd-party C extension is using this API, and some googling suggests that the API is barely used by any extensions. To get in trouble, C code has to use this API to target a thread that happens to be shutting down, and then suffer a sequence of unlikely timing accidents. It seems unlikely that code using this API *would* target a thread that's shutting down (e.g., it's more likely to target a thread that's *too* active <wink>), which may be why it hasn't been reported yet in real life. So it's OK by my if everyone pretends I didn't file the bug report until a month after 2.4 final was released; Guido may even be able to arrange to make that true. Of course it's fine to get very drunk regardless. From mulk at gmx.net Fri Nov 19 18:55:23 2004 From: mulk at gmx.net (Matthias Andreas Benkard) Date: Fri Nov 19 18:48:38 2004 Subject: [Python-Dev] Confusion about os.urandom() Message-ID: <1100886923.10228.7.camel@mulkomp.mst-plus> Hello there. I was just reading through http://www.python.org/2.4/highlights.html and noticed the following: os.urandom() has been added for systems that support a source of random data (entropy) At first, I was a bit stunned about the choice of name here. Why would anyone call this method urandom()? That confused me a bit, for, AFAICS, under Linux at least, /dev/random is the entropy pool and /dev/urandom is a PRNG (or rather, a source of random numbers which falls back to a PRNG when the entropy pool runs out of numbers). So I would not expect a method that is supposed to yield cryptographically secure random numbers to be called `urandom()'. Anyway, that might be non-intuitive (which is unusual in Python), but certainly not a severe bug. But then I noticed that if I rename /dev/urandom to something else (say, /dev/notrandomenough), os.urandom() (in Python 2.4b2) raises a NotImplementedError: Python 2.4b2 (#2, Nov 11 2004, 23:44:54) [GCC 3.3.5 (Debian 1:3.3.5-2)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import os >>> os.urandom(1) Traceback (most recent call last): File "<stdin>", line 1, in ? File "/usr/lib/python2.4/os.py", line 693, in urandom raise NotImplementedError("/dev/urandom (or equivalent) not found") NotImplementedError: /dev/urandom (or equivalent) not found >>> Now I'm really confused. Does os.urandom() use /dev/urandom under Linux? Either my brain has got a serious bug that makes me misunderstand the random(4) manpage, which states that "[w]hen read, /dev/urandom device will return as many bytes as are requested. As a result, if there is not sufficient entropy in the entropy pool, the returned values are theoretically vulnerable to a cryptographic attack on the algorithms used by the driver", or the implementation of os.urandom() is severely flawed. (BTW, I just checked Python 2.4c1. Judging from the source code, it seems to have the same behaviour.) That's what help(os.urandom) says: urandom(n) -> str Return a string of n random bytes suitable for cryptographic use. So it should be using /dev/random rather than /dev/urandom, shouldn't it? - Matthias A. Benkard -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Dies ist ein digital signierter Nachrichtenteil Url : http://mail.python.org/pipermail/python-dev/attachments/20041119/1a293b22/attachment.pgp From FBatista at uniFON.com.ar Fri Nov 19 20:02:55 2004 From: FBatista at uniFON.com.ar (Batista, Facundo) Date: Fri Nov 19 20:04:48 2004 Subject: [Python-Dev] Misinformation in highlights webpage Message-ID: <A128D751272CD411BC9200508BC2194D053C7C25@escpl.tcp.com.ar> In http://www.python.org/2.4/highlights.html it presents decimal between the "New or upgraded built-ins", while it should talk about it in "New or upgraded modules and packages". Should I post a bug or this is enough? (never know what to do regarding webpages). Thanks! Facundo Batista Desarrollo de Red fbatista@unifon.com.ar (54 11) 5130-4643 Cel: 15 5097 5024 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20041119/0f3d7c2a/attachment.html From irmen at xs4all.nl Fri Nov 19 20:16:55 2004 From: irmen at xs4all.nl (Irmen de Jong) Date: Fri Nov 19 20:16:56 2004 Subject: [Python-Dev] RELEASED Python 2.4, release candidate 1 In-Reply-To: <200411191816.52309.anthony@interlink.com.au> References: <200411191816.52309.anthony@interlink.com.au> Message-ID: <419E46A7.1080501@xs4all.nl> Anthony Baxter wrote: > Notable changes in rc1 include a handful of bug fixes, including a > thread shutdown race bug. See either the highlights, the What's New in > Python 2.4, or the detailed NEWS file -- all available from the Python > 2.4 webpage. I'm looking in the NEWS file, but can't seem to locate the fixed bugs/patches mentioned in the status page of the last bugday: http://www.python.org/moin/PythonBugDayStatus Are they left out? Or simply not mentioned? --Irmen de Jong From anthony at interlink.com.au Fri Nov 19 20:41:41 2004 From: anthony at interlink.com.au (Anthony Baxter) Date: Fri Nov 19 20:41:52 2004 Subject: [Python-Dev] RELEASED Python 2.4, release candidate 1 In-Reply-To: <419E46A7.1080501@xs4all.nl> References: <200411191816.52309.anthony@interlink.com.au> <419E46A7.1080501@xs4all.nl> Message-ID: <200411200641.41478.anthony@interlink.com.au> On Saturday 20 November 2004 06:16, Irmen de Jong wrote: > I'm looking in the NEWS file, but can't seem to locate the > fixed bugs/patches mentioned in the status page of the last > bugday: http://www.python.org/moin/PythonBugDayStatus > Are they left out? Or simply not mentioned? Beats me. Did whoever do the checkin add entries in the NEWS file? If not, it's unlikely that they will have been added. I long ago gave up on trying to trawl back through CVS changelogs and match them to NEWS entries - life's too short. Anthony From tjreedy at udel.edu Fri Nov 19 21:06:27 2004 From: tjreedy at udel.edu (Terry Reedy) Date: Fri Nov 19 21:06:22 2004 Subject: [Python-Dev] Re: syntactic shortcut - unpack to variably sized list References: <000c01c4c838$77bb3da0$0d00a8c0@MAINFRAME><864d3709041117184370f63c1b@mail.gmail.com><419C8276.5030809@iinet.net.au><864d370904111803561c650864@mail.gmail.com><419DCB19.5000609@iinet.net.au> <864d37090411190724440dfe38@mail.gmail.com> Message-ID: <cnljnp$8r8$1@sea.gmane.org> "Carlos Ribeiro" <carribeiro@gmail.com> wrote in message news:864d37090411190724440dfe38@mail.gmail.com... > For more than a few arguments, it seems to be silly to require the > user to write it as: > a,b,c,d,e = t[0],t[1],t[2],t[3],t[4:] and also impossible. >>> t=range(10) >>> a,b,c,d=t[:4]; e=t[4:] >>> a,b,c,d,e (0, 1, 2, 3, [4, 5, 6, 7, 8, 9]) but this does require that return values be captured in a temporary first. This issue has come up enough that a PEP would be useful (if there isn't one already). Terry J. Reedy From martin at v.loewis.de Fri Nov 19 21:46:47 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri Nov 19 21:46:47 2004 Subject: [Python-Dev] TRUNK UNFROZEN In-Reply-To: <200411192203.52490.anthony@interlink.com.au> References: <200411192203.52490.anthony@interlink.com.au> Message-ID: <419E5BB7.1040902@v.loewis.de> Anthony Baxter wrote: > Thanks to Tim's latest discovery, it looks like we might need an rc2 - > I might try to aim for this in a week's time (assuming that's ok with > Fred and Martin) and try to keep things moving along to 2.4 final. That's fine with me. Regards, Martin From foom at fuhm.net Fri Nov 19 21:48:19 2004 From: foom at fuhm.net (James Y Knight) Date: Fri Nov 19 21:48:22 2004 Subject: [Python-Dev] print "%X" % id(object()) not so nice Message-ID: <582E4A36-3A6C-11D9-AA57-000A95A50FB2@fuhm.net> I think id() should never be returning a negative number. Both these behaviors are poor: In 2.3: >>> print "%X" % id(o) __main__:1: FutureWarning: %u/%o/%x/%X of negative int will return a signed string in Python 2.4 and up A5F48198 In 2.4: >>> print "%X" %id(o) -5FC84D08 Pointers are conventionally never treated or printed as signed. In 2.3 and before, it usually ended up okay, besides the warning, because "%X" had broken behavior. In 2.4, now it's ending up doing the wrong thing and printing a confusing value. I propose that id() always return a positive value. This means that it will sometimes have to return a long instead of an int, but, it already does that under some circumstances on some architectures. Comments? James From martin at v.loewis.de Fri Nov 19 21:59:06 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri Nov 19 21:59:06 2004 Subject: [Python-Dev] Confusion about os.urandom() In-Reply-To: <1100886923.10228.7.camel@mulkomp.mst-plus> References: <1100886923.10228.7.camel@mulkomp.mst-plus> Message-ID: <419E5E9A.7080502@v.loewis.de> Matthias Andreas Benkard wrote: > At first, I was a bit stunned about the choice of name here. Why would > anyone call this method urandom()? That confused me a bit, for, AFAICS, > under Linux at least, /dev/random is the entropy pool and /dev/urandom > is a PRNG (or rather, a source of random numbers which falls back to a > PRNG when the entropy pool runs out of numbers). That is not true. It doesn't first exhaust the pool and then falls back to PRNG. Instead, it gradually moves to a PRNG, depending on the amount of entropy in the pool. The values returned are still cryptographically secure, except in purely theoretical cases (where a lot of entropy is drawn from random or urandom, and nothing is filled in). > Now I'm really confused. Does os.urandom() use /dev/urandom under Linux? Yes, it does. > That's what help(os.urandom) says: > > urandom(n) -> str > Return a string of n random bytes suitable for cryptographic use. > > So it should be using /dev/random rather than /dev/urandom, shouldn't > it? No, it shouldn't. /dev/random may block, which os.urandom() will not. The name urandom deliberately tells users that there is a theoretical flaw (which is practically irrelevant). If users cannot stand the theoretical flaw, they need to use /dev/random (which also has theoretical flaws that just happen to be even less practically relevant). In that case a) they have to accept that reading /dev/random might block indefinitely, and b) their code will become more system-dependent. Regards, Martin From martin at v.loewis.de Fri Nov 19 22:02:15 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri Nov 19 22:02:16 2004 Subject: [Python-Dev] print "%X" % id(object()) not so nice In-Reply-To: <582E4A36-3A6C-11D9-AA57-000A95A50FB2@fuhm.net> References: <582E4A36-3A6C-11D9-AA57-000A95A50FB2@fuhm.net> Message-ID: <419E5F57.10509@v.loewis.de> James Y Knight wrote: > Comments? This cannot be implemented for 2.4. For 2.5, it is reasonable; feel free to submit a patch. Regards, Martin From jason at tishler.net Sat Nov 20 00:27:33 2004 From: jason at tishler.net (Jason Tishler) Date: Sat Nov 20 00:25:04 2004 Subject: [Python-Dev] Current CVS, Cygwin and "make test" In-Reply-To: <419E0B4E.7040109@iinet.net.au> References: <419E0B4E.7040109@iinet.net.au> Message-ID: <20041119232733.GB1680@tishler.net> Nick, On Sat, Nov 20, 2004 at 01:03:42AM +1000, Nick Coghlan wrote: > I only got the regression tests working *at all* today, after doing > 'find . -name "\.dll" | rebaseall -T -' from my python/dist/src > directory to get rid of the Cygwin remap errors when loading DLL's. The above is expected behavior, but see below... > I figured I'd ask the folks here for ideas (is Jason still around?), > before trying to ask the question from the Cygwin end. Yes, I'm still around. :,) What Cygwin version are you using? 1.5.12? FWIW, I got the following results when running Python-2.4b1's normal (i.e., no extras) regression test: $ ./python.exe -E -tt ./Lib/test/regrtest.py -l ... 252 tests OK. 2 tests failed: test_subprocess test_tcl ... The above results was under a pre-1.5.12 Cygwin CVS on XP Pro SP1: $ uname -a CYGWIN_NT-5.1 gelpdevjt022 1.5.12(0.116/4/2) 2004-10-28 07:28 ... Amazingly, I did *not* need to run rebase like I normal do! Please let me know if you need help tracking down Cygwin related problems. Thanks, Jason -- PGP/GPG Key: http://www.tishler.net/jason/pubkey.asc or key servers Fingerprint: 7A73 1405 7F2B E669 C19D 8784 1AFD E4CC ECF4 8EF6 From tjreedy at udel.edu Sat Nov 20 01:38:21 2004 From: tjreedy at udel.edu (Terry Reedy) Date: Sat Nov 20 01:38:18 2004 Subject: [Python-Dev] Re: print "%X" % id(object()) not so nice References: <582E4A36-3A6C-11D9-AA57-000A95A50FB2@fuhm.net> Message-ID: <cnm3lj$9nj$1@sea.gmane.org> "James Y Knight" <foom@fuhm.net> wrote in message news:582E4A36-3A6C-11D9-AA57-000A95A50FB2@fuhm.net... >[snip] >I propose that id() always return a positive value ... > Comments? 1. CPython intentionally searches builtins afters globals and pre-imports the former as __builtins__ just so one can wrap a builtin to modify its apparent behavior for personal needs. >>> def id(o, max = 2**32): ... i = __builtins__.id(o) ... return (i < 0) and (max - i) or i ... >>> id(a) 9158224 # only semi-tested because I don't have negatives to test and can't force 2. Given that, why bother changing the language for what must be an esoteric need (to formattedly print and view ids)? The id of an object is constant and unique with respect to contemporaneous objects but, for CPython, definitely not with respect to objects with non-overlapping lifetimes. (Newbies often get tripped by the last fact.). >From the viewpoint of Python, ids have no meaning and are only useful for identity comparision. For this purpose, arbitrary strings would have worked as well as integers. For convenience, CPython uses the apparent address stored as an int. But this is strictly an implementation detail. On modern systems, that 'address' is, I believe, a process-specific virtual address which the hardware memory management system maps the hidden real address -- which is the only reason why systems with less than 2**31 memory can have addresses at or above 2**31 to become negative ints. Terry J. Reedy From Scott.Daniels at Acm.Org Sat Nov 20 01:52:55 2004 From: Scott.Daniels at Acm.Org (Scott David Daniels) Date: Sat Nov 20 01:51:41 2004 Subject: [Python-Dev] Re: print "%X" % id(object()) not so nice In-Reply-To: <cnm3lj$9nj$1@sea.gmane.org> References: <582E4A36-3A6C-11D9-AA57-000A95A50FB2@fuhm.net> <cnm3lj$9nj$1@sea.gmane.org> Message-ID: <cnm4ek$b4v$1@sea.gmane.org> Terry Reedy wrote: >>>>def id(o, max = 2**32): > > ... i = __builtins__.id(o) > ... return (i < 0) and (max - i) or i > ... > This code doesn't need an extra parameter: def id(o): return (sys.maxint * 2 + 1) & __builtins__.id(o) -- -- Scott David Daniels Scott.Daniels@Acm.Org From bob at redivi.com Sat Nov 20 01:55:08 2004 From: bob at redivi.com (Bob Ippolito) Date: Sat Nov 20 01:55:53 2004 Subject: [Python-Dev] Re: print "%X" % id(object()) not so nice In-Reply-To: <cnm3lj$9nj$1@sea.gmane.org> References: <582E4A36-3A6C-11D9-AA57-000A95A50FB2@fuhm.net> <cnm3lj$9nj$1@sea.gmane.org> Message-ID: <D3128502-3A8E-11D9-925A-000A9567635C@redivi.com> On Nov 20, 2004, at 2:38 AM, Terry Reedy wrote: > > "James Y Knight" <foom@fuhm.net> wrote in message > news:582E4A36-3A6C-11D9-AA57-000A95A50FB2@fuhm.net... >> [snip] >> I propose that id() always return a positive value ... >> Comments? > > 1. CPython intentionally searches builtins afters globals and > pre-imports > the former as __builtins__ just so one can wrap a builtin to modify its > apparent behavior for personal needs. > >>>> def id(o, max = 2**32): > ... i = __builtins__.id(o) > ... return (i < 0) and (max - i) or i > ... >>>> id(a) > 9158224 > # only semi-tested because I don't have negatives to test and can't > force > > 2. Given that, why bother changing the language for what must be an > esoteric need (to formattedly print and view ids)? > > The id of an object is constant and unique with respect to > contemporaneous > objects but, for CPython, definitely not with respect to objects with > non-overlapping lifetimes. (Newbies often get tripped by the last > fact.). >> From the viewpoint of Python, ids have no meaning and are only useful >> for > identity comparision. For this purpose, arbitrary strings would have > worked as well as integers. > > For convenience, CPython uses the apparent address stored as an int. > But > this is strictly an implementation detail. On modern systems, that > 'address' is, I believe, a process-specific virtual address which the > hardware memory management system maps the hidden real address -- > which is > the only reason why systems with less than 2**31 memory can have > addresses > at or above 2**31 to become negative ints. The problem, more than anything else, is the following behavior that can happen during a random __repr__ or repr-like-function if the object happens to have a certain address range: - (Python 2.3) You get an unexpected and unwanted warning but expected output anyway - (Python 2.4) You get a repr with a strange looking negative hex number (0x-FF0102) Neither of these are fatal, of course, it's just annoying.. I find the Python 2.3 behavior more obnoxious than Python 2.4's, personally. FYI, I have also encountered this "problem" this week on a Powerbook G4 w/ only 1GB physical memory on both Python 2.3 and 2.4. I'm at the PyPy sprint, and a lot of the tools we are using make use of repr. Fortunately we have control over all of this code, so I checked in a workaround that makes sure a sane value was passed to the hex formatter: import sys HUGEINT = (sys.maxint + 1L) * 2L def uid(obj): """ Return the id of an object as an unsigned number so that its hex representation makes sense """ rval = id(obj) if rval < 0: rval += HUGEINT return rval -bob From ncoghlan at iinet.net.au Sat Nov 20 03:28:50 2004 From: ncoghlan at iinet.net.au (Nick Coghlan) Date: Sat Nov 20 03:28:56 2004 Subject: [Python-Dev] Current CVS, Cygwin and "make test" In-Reply-To: <20041119232733.GB1680@tishler.net> References: <419E0B4E.7040109@iinet.net.au> <20041119232733.GB1680@tishler.net> Message-ID: <419EABE2.9040804@iinet.net.au> Jason Tishler wrote: > Nick, > > On Sat, Nov 20, 2004 at 01:03:42AM +1000, Nick Coghlan wrote: > >>I only got the regression tests working *at all* today, after doing >>'find . -name "\.dll" | rebaseall -T -' from my python/dist/src >>directory to get rid of the Cygwin remap errors when loading DLL's. > > The above is expected behavior, but see below... I thought so - it was your advice that Google turned up to tell me how to fix it :) >>I figured I'd ask the folks here for ideas (is Jason still around?), >>before trying to ask the question from the Cygwin end. > > Yes, I'm still around. :,) > > What Cygwin version are you using? 1.5.12? 1.5.11 (also on XP Pro SP1) $ uname -a CYGWIN_NT-5.1 pseudopolis 1.5.11(0.116/4/2) 2004-09-04 23:17 ... I'll try updating all the Cygwin packages and see where that gets me. <that didn't take long> Now it's hanging in configure (and the m4 interpreter was one of the things that got updated). *sigh* I need to wake up more before I dig into this. If you (or anyone else) could check that the unit tests work on Cygwin for you, that would make me more certain that this is just a problem with my own Cygwin setup, and not related to anything in 2.4c1 that wasn't in 2.4b1. Cheers, Nick. -- Nick Coghlan | Brisbane, Australia Email: ncoghlan@email.com | Mobile: +61 409 573 268 From ncoghlan at iinet.net.au Sat Nov 20 03:29:50 2004 From: ncoghlan at iinet.net.au (Nick Coghlan) Date: Sat Nov 20 03:29:55 2004 Subject: [Python-Dev] Re: syntactic shortcut - unpack to variably sized list In-Reply-To: <cnljnp$8r8$1@sea.gmane.org> References: <000c01c4c838$77bb3da0$0d00a8c0@MAINFRAME><864d3709041117184370f63c1b@mail.gmail.com><419C8276.5030809@iinet.net.au><864d370904111803561c650864@mail.gmail.com><419DCB19.5000609@iinet.net.au> <864d37090411190724440dfe38@mail.gmail.com> <cnljnp$8r8$1@sea.gmane.org> Message-ID: <419EAC1E.6020902@iinet.net.au> Terry Reedy wrote: > "Carlos Ribeiro" <carribeiro@gmail.com> wrote in message > news:864d37090411190724440dfe38@mail.gmail.com... > >>For more than a few arguments, it seems to be silly to require the >>user to write it as: >>a,b,c,d,e = t[0],t[1],t[2],t[3],t[4:] > > > and also impossible. > >>>>t=range(10) >>>>a,b,c,d=t[:4]; e=t[4:] >>>>a,b,c,d,e > > (0, 1, 2, 3, [4, 5, 6, 7, 8, 9]) Hmm. . . _>>>t = range(10) _>>>a, b, c, d, e = t[:4] + [t[4:]] _>>>a, b, c, d, e _(0, 1, 2, 3, [4, 5, 6, 7, 8, 9]) So this actually can scale (sort of) to longer tuples. Anyway, I don't have any real objection to iunpack. If it was designed to work on any iterator (return the first few elements, then return the partially consumed iterator), I'd be +1 (since, as Carlos pointed out, slicing doesn't work for arbitrary iterators). Something like: _>>>def iunpack(itr, numitems, defaultitem=None): _... for i in range(numitems): _... try: _... yield itr.next() _... except StopIteration: _... yield defaultitem _... yield itr _>>> g = (x for x in range(10)) _>>> a, b, c, d, e = iunpack(g, 4) _>>> a, b, c, d, e _(0, 1, 2, 3, <generator object at 0xa0cd02c>) _>>> a, b, c, d, e = iunpack(g, 4) _>>> a, b, c, d, e _(4, 5, 6, 7, <generator object at 0xa0cd02c>) _>>> a, b, c, d, e = iunpack(g, 4) _>>> a, b, c, d, e _(8, 9, None, None, <generator object at 0xa0cd02c>) Cheers, Nick. I think we're now to the stage of violently agreeing with each other. . . -- Nick Coghlan | Brisbane, Australia Email: ncoghlan@email.com | Mobile: +61 409 573 268 From tjreedy at udel.edu Sat Nov 20 03:53:08 2004 From: tjreedy at udel.edu (Terry Reedy) Date: Sat Nov 20 03:53:07 2004 Subject: [Python-Dev] Re: print "%X" % id(object()) not so nice References: <582E4A36-3A6C-11D9-AA57-000A95A50FB2@fuhm.net> <cnm3lj$9nj$1@sea.gmane.org> Message-ID: <cnmbia$msh$1@sea.gmane.org> "Terry Reedy" <tjreedy@udel.edu> wrote in message news:cnm3lj$9nj$1@sea.gmane.org... > ... return (i < 0) and (max - i) or i > # only semi-tested because I don't have negatives to test and can't force whoops, untested negative case should be max + i (or max-abs(i) as I was thinking) From tjreedy at udel.edu Sat Nov 20 04:03:15 2004 From: tjreedy at udel.edu (Terry Reedy) Date: Sat Nov 20 04:03:14 2004 Subject: [Python-Dev] Re: Re: print "%X" % id(object()) not so nice References: <582E4A36-3A6C-11D9-AA57-000A95A50FB2@fuhm.net><cnm3lj$9nj$1@sea.gmane.org> <D3128502-3A8E-11D9-925A-000A9567635C@redivi.com> Message-ID: <cnmc59$npt$1@sea.gmane.org> "Bob Ippolito" <bob@redivi.com> wrote in message news:D3128502-3A8E-11D9-925A-000A9567635C@redivi.com... > The problem, more than anything else, is the following behavior that can > happen during a random __repr__ or repr-like-function if the object > happens to have a certain address range: > > - (Python 2.3) You get an unexpected and unwanted warning but expected > output anyway > - (Python 2.4) You get a repr with a strange looking negative hex number > (0x-FF0102) > > Neither of these are fatal, of course, it's just annoying.. I find the > Python 2.3 behavior more obnoxious than Python 2.4's, personally. Non-CS users probably find *all* hex numbers a little strange looking. If CPython were to simply print ids as decimal integers, instead of being fancy with hex 'addresses' there would have been no warnings and no change ;-). Is the absolute hex value ever of any use? If so, how often? Terry J. Reedy From ncoghlan at iinet.net.au Sat Nov 20 04:41:19 2004 From: ncoghlan at iinet.net.au (Nick Coghlan) Date: Sat Nov 20 04:41:28 2004 Subject: [Python-Dev] Re: syntactic shortcut - unpack to variably sized list In-Reply-To: <cnljnp$8r8$1@sea.gmane.org> References: <000c01c4c838$77bb3da0$0d00a8c0@MAINFRAME><864d3709041117184370f63c1b@mail.gmail.com><419C8276.5030809@iinet.net.au><864d370904111803561c650864@mail.gmail.com><419DCB19.5000609@iinet.net.au> <864d37090411190724440dfe38@mail.gmail.com> <cnljnp$8r8$1@sea.gmane.org> Message-ID: <419EBCDF.1070400@iinet.net.au> Terry Reedy wrote: > This issue has come up enough that a PEP would be useful (if there isn't > one already). Nope, I had a look. I've got a draft one attached, though - it combines rejecting the syntax change with adding the other changes Carlos and I have suggested. If people think it's worth pursuing, I'll fire it off to the PEP editors. Cheers, Nick. -- Nick Coghlan | Brisbane, Australia Email: ncoghlan@email.com | Mobile: +61 409 573 268 -------------- next part -------------- PEP: XXX Title: Unpacking sequence elements using tuple assignment Version: $Revision: 1.4 $ Last-Modified: $Date: 2003/09/22 04:51:50 $ Author: Nick Coghlan <ncoghlan@email.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 20-Nov-2004 Python-Version: 2.5 Post-History: 20-Nov-2004 Abstract ======== This PEP addresses the repeated proposal that "a, b, *c = some_sequence" be permissible Python syntax. The PEP does NOT propose that this syntax be allowed. In fact, acceptance of this PEP will indicate agreement that the above syntax will NOT become part of Python. However, this PEP does suggest two additions that will hopefully reduce the demand for the above syntax: - modify ``list.pop`` and ``array.pop`` to also accept slice objects as the index argument - add an ``itertools.iunpack`` function that accepts an arbitrary iterator, yields a specified number of elements from it, and then yields the partially consumed iterator. Rationale ========= The proposal to extend the tuple unpacking assignment syntax has now been posted to python-dev on 3 separate occasions, and rejected on all 3 occasions.[1]_ The subsequent discussion on the most recent occasion yielded the two suggestions documented in this PEP (modifying ``list.pop`` was suggested by the PEP author, the ``itertools.iunpack`` function was suggested by Carlos Ribeiro). Modifying ``list.pop`` to accept slice objects as well as integers brings it in line with the standard sequence subscripting methods (``__getitem__``, ``__setitem__``, ``__delitem__``). It also makes ``list.pop`` consistent with its current documentation (the Python code given as equivalent to ``list.pop`` accepts slice objects, but ``list.pop`` does not). The modification of ``array.pop`` is mainly for consistency with the new behaviour of ``list.pop``. The ``itertools.iunpack`` function has the advantage of working for arbitrary iterators, not just lists or arrays. However, it always yields copies of the elements, while the ``pop`` methods are able to avoid unnecessary copying. Proposed Semantics ================== ``list.pop`` would be updated to conform to its documentation as being equivalent to:: x = L[i] del L[i] return x In Python 2.4, the above equivalence does not hold if ``i`` is a slice object. ``array.pop`` would be updated similarly. ``itertools.iunpack`` would be equivalent to the following:: def iunpack(itr, numitems, defaultitem=None): for i in range(numitems): try: yield itr.next() except StopIteration: yield defaultitem yield itr Reference Implementation ======================== As yet, no reference implementation is available for either part of the proposal. Open Issues =========== - Should ``itertools.iunpack`` call ``iter()`` on its first argument? References ========== .. [1] python-dev archives of tuple unpacking discussions (http://mail.python.org/pipermail/python-dev/2002-November/030349.html) (http://mail.python.org/pipermail/python-dev/2004-August/046684.html) (http://mail.python.org/pipermail/python-dev/2004-November/049895.html) Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 End: From bob at redivi.com Sat Nov 20 08:48:15 2004 From: bob at redivi.com (Bob Ippolito) Date: Sat Nov 20 08:49:01 2004 Subject: [Python-Dev] Re: Re: print "%X" % id(object()) not so nice In-Reply-To: <cnmc59$npt$1@sea.gmane.org> References: <582E4A36-3A6C-11D9-AA57-000A95A50FB2@fuhm.net><cnm3lj$9nj$1@sea.gmane.org> <D3128502-3A8E-11D9-925A-000A9567635C@redivi.com> <cnmc59$npt$1@sea.gmane.org> Message-ID: <894CD7B9-3AC8-11D9-925A-000A9567635C@redivi.com> On Nov 20, 2004, at 5:03 AM, Terry Reedy wrote: > > "Bob Ippolito" <bob@redivi.com> wrote in message > news:D3128502-3A8E-11D9-925A-000A9567635C@redivi.com... >> The problem, more than anything else, is the following behavior that >> can >> happen during a random __repr__ or repr-like-function if the object >> happens to have a certain address range: >> >> - (Python 2.3) You get an unexpected and unwanted warning but expected >> output anyway >> - (Python 2.4) You get a repr with a strange looking negative hex >> number >> (0x-FF0102) >> >> Neither of these are fatal, of course, it's just annoying.. I find the >> Python 2.3 behavior more obnoxious than Python 2.4's, personally. > > Non-CS users probably find *all* hex numbers a little strange looking. > If > CPython were to simply print ids as decimal integers, instead of being > fancy with hex 'addresses' there would have been no warnings and no > change > ;-). Is the absolute hex value ever of any use? If so, how often? It makes it quite easy to match pdb output with gdb output! :) -bob From jason at tishler.net Sat Nov 20 16:06:37 2004 From: jason at tishler.net (Jason Tishler) Date: Sat Nov 20 16:05:00 2004 Subject: [Python-Dev] Current CVS, Cygwin and "make test" In-Reply-To: <419EABE2.9040804@iinet.net.au> References: <419E0B4E.7040109@iinet.net.au> <20041119232733.GB1680@tishler.net> <419EABE2.9040804@iinet.net.au> Message-ID: <20041120150637.GA3024@tishler.net> Nick, On Sat, Nov 20, 2004 at 12:28:50PM +1000, Nick Coghlan wrote: > Jason Tishler wrote: > >What Cygwin version are you using? 1.5.12? > > 1.5.11 (also on XP Pro SP1) Cygwin 1.5.11 has some issues (e.g., pthreads), which is why I skipped that version... :,) > I'll try updating all the Cygwin packages and see where that gets me. > > <that didn't take long> Did you upgrade Cygwin to 1.5.12? > Now it's hanging in configure (and the m4 interpreter was one of the > things that got updated). *sigh* I need to wake up more before I dig > into this. The above is strange, but the m4 package was just taken over by another maintainer. > If you (or anyone else) could check that the unit tests work on Cygwin > for you, that would make me more certain that this is just a problem > with my own Cygwin setup, and not related to anything in 2.4c1 that > wasn't in 2.4b1. I just upgraded one of my machines to Cygwin 1.5.12, but I did not upgrade the other packages. I get the following results: failures: test_shutil (passed under older Cygwin and/or Python versions) test_subprocess (new test) test_tcl (new test) hangs: test_threaded_import (passed under older Cygwin and/or Python versions) I will investigate the above and report back. Thanks, Jason -- PGP/GPG Key: http://www.tishler.net/jason/pubkey.asc or key servers Fingerprint: 7A73 1405 7F2B E669 C19D 8784 1AFD E4CC ECF4 8EF6 From tjreedy at udel.edu Sat Nov 20 16:12:37 2004 From: tjreedy at udel.edu (Terry Reedy) Date: Sat Nov 20 16:12:32 2004 Subject: [Python-Dev] Re: Re: Re: print "%X" % id(object()) not so nice References: <582E4A36-3A6C-11D9-AA57-000A95A50FB2@fuhm.net><cnm3lj$9nj$1@sea.gmane.org><D3128502-3A8E-11D9-925A-000A9567635C@redivi.com><cnmc59$npt$1@sea.gmane.org> <894CD7B9-3AC8-11D9-925A-000A9567635C@redivi.com> Message-ID: <cnnmsr$vc2$1@sea.gmane.org> "Bob Ippolito" <bob@redivi.com> wrote in message news:894CD7B9- > On Nov 20, 2004, at 5:03 AM, Terry Reedy wrote: >> Is the absolute hex value ever of any use? If so, how often? > It makes it quite easy to match pdb output with gdb output! :) Ah, the missing use case, which you and the OP probably took for granted. I, on the other hand, having never used either, find the difference in printed ids in >>> def f(): pass ... >>> f, id(f) (<function f at 0x00868158>, 8814936) at least mildly disturbing. Do you only need to do such matching for complex objects that get the <type name at 0x########> representation? Terry J. Reedy From mwh at python.net Sat Nov 20 16:41:24 2004 From: mwh at python.net (Michael Hudson) Date: Sat Nov 20 16:41:27 2004 Subject: [Python-Dev] Re: Re: Re: print "%X" % id(object()) not so nice In-Reply-To: <cnnmsr$vc2$1@sea.gmane.org> (Terry Reedy's message of "Sat, 20 Nov 2004 10:12:37 -0500") References: <582E4A36-3A6C-11D9-AA57-000A95A50FB2@fuhm.net> <cnm3lj$9nj$1@sea.gmane.org> <D3128502-3A8E-11D9-925A-000A9567635C@redivi.com> <cnmc59$npt$1@sea.gmane.org> <894CD7B9-3AC8-11D9-925A-000A9567635C@redivi.com> <cnnmsr$vc2$1@sea.gmane.org> Message-ID: <2mekiowmsb.fsf@starship.python.net> "Terry Reedy" <tjreedy@udel.edu> writes: > I, on the other hand, having never used either, find the difference in > printed ids in > >>>> def f(): pass > ... >>>> f, id(f) > (<function f at 0x00868158>, 8814936) > > at least mildly disturbing. Do you only need to do such matching for > complex objects that get the <type name at 0x########> representation? This hardly seems worth discussing :) It's a pointer. Pointers are printed in hex. It's Just The Way It Is. I don't know why. Actually, the "0x00868158" above is produced by C's %p format operator. So, in fact, ANSI C is probably why it is The Way It Is. Cheers, mwh -- Remember - if all you have is an axe, every problem looks like hours of fun. -- Frossie -- http://home.xnet.com/~raven/Sysadmin/ASR.Quotes.html From carribeiro at gmail.com Sat Nov 20 16:55:38 2004 From: carribeiro at gmail.com (Carlos Ribeiro) Date: Sat Nov 20 16:55:41 2004 Subject: [Python-Dev] Re: syntactic shortcut - unpack to variably sized list In-Reply-To: <419EBCDF.1070400@iinet.net.au> References: <000c01c4c838$77bb3da0$0d00a8c0@MAINFRAME> <864d3709041117184370f63c1b@mail.gmail.com> <419C8276.5030809@iinet.net.au> <864d370904111803561c650864@mail.gmail.com> <419DCB19.5000609@iinet.net.au> <864d37090411190724440dfe38@mail.gmail.com> <cnljnp$8r8$1@sea.gmane.org> <419EBCDF.1070400@iinet.net.au> Message-ID: <864d370904112007554e3720a2@mail.gmail.com> On Sat, 20 Nov 2004 13:41:19 +1000, Nick Coghlan <ncoghlan@iinet.net.au> wrote: > Terry Reedy wrote: > > This issue has come up enough that a PEP would be useful (if there isn't > > one already). > > Nope, I had a look. I've got a draft one attached, though - it combines > rejecting the syntax change with adding the other changes Carlos and I have > suggested. If people think it's worth pursuing, I'll fire it off to the PEP editors. I may be wrong (and if that's the case, I would like to be politely educated on why), but isn't it already on a PEP-worthy point? IOW, if people are not interested, then the PEP will simply be rejected, which is a good thing, because it will at least document the case. I also believe that the pre-PEP should be posted to the main Python list, where it may be beaten to death & flamed by a larger audience. I am willing to do it myself, but I assume that is more polite on my part if I ask you to do it :-) <PEP boilerplate & rationale snipped> > Reference Implementation > ======================== > > As yet, no reference implementation is available for either part of the proposal. I though the iunpack() code in the previous section would be acceptable as a 'reference implementation'. > Open Issues > =========== > > - Should ``itertools.iunpack`` call ``iter()`` on its first argument? +1 > References > ========== > > .. [1] python-dev archives of tuple unpacking discussions > (http://mail.python.org/pipermail/python-dev/2002-November/030349.html) > (http://mail.python.org/pipermail/python-dev/2004-August/046684.html) > (http://mail.python.org/pipermail/python-dev/2004-November/049895.html) > > Copyright > ========= > > This document has been placed in the public domain. > > .. > Local Variables: > mode: indented-text > indent-tabs-mode: nil > sentence-end-double-space: t > fill-column: 70 > End: > > > _______________________________________________ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/carribeiro%40gmail.com > > > -- Carlos Ribeiro Consultoria em Projetos blog: http://rascunhosrotos.blogspot.com blog: http://pythonnotes.blogspot.com mail: carribeiro@gmail.com mail: carribeiro@yahoo.com From tim.peters at gmail.com Sat Nov 20 17:21:28 2004 From: tim.peters at gmail.com (Tim Peters) Date: Sat Nov 20 17:21:31 2004 Subject: [Python-Dev] Re: Re: Re: print "%X" % id(object()) not so nice In-Reply-To: <2mekiowmsb.fsf@starship.python.net> References: <582E4A36-3A6C-11D9-AA57-000A95A50FB2@fuhm.net> <cnm3lj$9nj$1@sea.gmane.org> <D3128502-3A8E-11D9-925A-000A9567635C@redivi.com> <cnmc59$npt$1@sea.gmane.org> <894CD7B9-3AC8-11D9-925A-000A9567635C@redivi.com> <cnnmsr$vc2$1@sea.gmane.org> <2mekiowmsb.fsf@starship.python.net> Message-ID: <1f7befae04112008214f9d810c@mail.gmail.com> [Terry Reedy] >> I, on the other hand, having never used either, find the difference in >> printed ids in >> >> >>> def f(): pass >> >>> f, id(f) >> (<function f at 0x00868158>, 8814936) >> >> at least mildly disturbing. Do you only need to do such matching for >> complex objects that get the <type name at 0x########> >> representation? [Michael Hudson] > This hardly seems worth discussing :) Then it's a topic for me <wink>! > It's a pointer. Pointers are printed in hex. It's Just The Way It > Is. I don't know why. > > Actually, the "0x00868158" above is produced by C's %p format > operator. So, in fact, ANSI C is probably why it is The Way It Is. repr starts with %p, but %p is ill-defined, so Python goes on to ensure the result starts with "0x". C doesn't even say that %p produces hex digits, but all C systems we know of do(*), so Python doesn't try to force that part. As to "why hex?", it's for low-level debugging. For example, stack, register and memory dumps for binary machines almost always come in some power-of-2 base, usually hex, and searching for a stored address is much easier if it's shown in the same base. OTOH, id(Q) promises to return an integer that won't be the same as the id() of any other object over Q's lifetime. CPython returns Q's memory address, but CPython never moves objects in memory, so CPython can get away with returning the address. Jython does something very different for id(), because it must -- the Java VM may move an object in memory. Python doesn't promise to return a postive integer for id(), although it may have been nicer if it did. It's dangerous to change that now, because some code does depend on the "32 bit-ness as a signed integer" accident of CPython's id() implementation on 32-bit machines. For example, code using struct.pack(), or code using one of ZODB's specialized int-key BTree types with id's as keys. Speaking of which, current ZODB has a positive_id() function, used to format id()'s in strings where a sign bit would get in the way. (*) The %p in some C's for early x86 systems, using "segment + offset" mode, stuck a colon "in the middle" of the pointer output, to visually separate the segment from the offset. The two parts were still shown in hex, though. From ncoghlan at iinet.net.au Sat Nov 20 17:47:49 2004 From: ncoghlan at iinet.net.au (Nick Coghlan) Date: Sat Nov 20 17:48:07 2004 Subject: [Python-Dev] Current CVS, Cygwin and "make test" In-Reply-To: <20041120150637.GA3024@tishler.net> References: <419E0B4E.7040109@iinet.net.au> <20041119232733.GB1680@tishler.net> <419EABE2.9040804@iinet.net.au> <20041120150637.GA3024@tishler.net> Message-ID: <419F7535.9050802@iinet.net.au> Jason Tishler wrote: > Did you upgrade Cygwin to 1.5.12? I did. However, the bash shell was just behaving strangely after that, as was the Cygwin installer itself. I tried to do a clean install, and the install program is now hanging. So I don't think we can take any error reports from my system seriously at this point. Something isn't happy :( Anyway, I won't be posting any more about this on py-dev, since I've established that *my* problems probably aren't Python related. Cheers, Nick. -- Nick Coghlan | Brisbane, Australia Email: ncoghlan@email.com | Mobile: +61 409 573 268 From ncoghlan at iinet.net.au Sat Nov 20 18:34:34 2004 From: ncoghlan at iinet.net.au (Nick Coghlan) Date: Sat Nov 20 18:34:41 2004 Subject: [Python-Dev] Re: syntactic shortcut - unpack to variably sizedlist In-Reply-To: <001501c4cf25$50aef780$e841fea9@oemcomputer> References: <001501c4cf25$50aef780$e841fea9@oemcomputer> Message-ID: <419F802A.5060209@iinet.net.au> Raymond Hettinger wrote: > Also, the pep would likely kill the remote, but non-zero chance of ever > getting the a,b,*c syntax. IOW, the pep may be a step backwards. Officially killing that syntax was actually one of the stated goals of the PEP. However, the PEP editors would be well within their rights to reject the draft as 'lacking focus'. I wouldn't be offended given the whole 20 minutes it took to write it :) Cheers, Nick. -- Nick Coghlan | Brisbane, Australia Email: ncoghlan@email.com | Mobile: +61 409 573 268 From ncoghlan at iinet.net.au Sat Nov 20 18:37:40 2004 From: ncoghlan at iinet.net.au (Nick Coghlan) Date: Sat Nov 20 18:37:46 2004 Subject: [Python-Dev] Re: syntactic shortcut - unpack to variably sized list In-Reply-To: <864d370904112007554e3720a2@mail.gmail.com> References: <000c01c4c838$77bb3da0$0d00a8c0@MAINFRAME> <864d3709041117184370f63c1b@mail.gmail.com> <419C8276.5030809@iinet.net.au> <864d370904111803561c650864@mail.gmail.com> <419DCB19.5000609@iinet.net.au> <864d37090411190724440dfe38@mail.gmail.com> <cnljnp$8r8$1@sea.gmane.org> <419EBCDF.1070400@iinet.net.au> <864d370904112007554e3720a2@mail.gmail.com> Message-ID: <419F80E4.1050009@iinet.net.au> Carlos Ribeiro wrote: > I may be wrong (and if that's the case, I would like to be politely > educated on why), but isn't it already on a PEP-worthy point? IOW, if > people are not interested, then the PEP will simply be rejected, which > is a good thing, because it will at least document the case. Good point - I've sent it to the PEP editors to request a number. > I also > believe that the pre-PEP should be posted to the main Python list, > where it may be beaten to death & flamed by a larger audience. I am > willing to do it myself, but I assume that is more polite on my part > if I ask you to do it :-) I'd wait to see what the PEP editors think, myself. I'm not entirely sure the idea is focused well enough to make a good PEP (I could argue either way, and I wrote the thing!). However, if you want to post it over there for discussion, feel free. >>Reference Implementation >>======================== >> >>As yet, no reference implementation is available for either part of the proposal. > > > I though the iunpack() code in the previous section would be > acceptable as a 'reference implementation'. Unfortunately, itertools is a C module :P >>Open Issues >>=========== >> >>- Should ``itertools.iunpack`` call ``iter()`` on its first argument? > > +1 We'll go with that then. . . y'know, I could have made that change before sending the PEP draft in. Ah well, we can change it later - I doubt the PEP will surive c.l.p. (or even py-dev) unscathed, anyway. Cheers, Nick. -- Nick Coghlan | Brisbane, Australia Email: ncoghlan@email.com | Mobile: +61 409 573 268 From carribeiro at gmail.com Sat Nov 20 19:14:59 2004 From: carribeiro at gmail.com (Carlos Ribeiro) Date: Sat Nov 20 19:15:03 2004 Subject: [Python-Dev] Re: syntactic shortcut - unpack to variably sizedlist In-Reply-To: <419F802A.5060209@iinet.net.au> References: <001501c4cf25$50aef780$e841fea9@oemcomputer> <419F802A.5060209@iinet.net.au> Message-ID: <864d37090411201014cbb0007@mail.gmail.com> On Sun, 21 Nov 2004 03:34:34 +1000, Nick Coghlan <ncoghlan@iinet.net.au> wrote: > Raymond Hettinger wrote: > > Also, the pep would likely kill the remote, but non-zero chance of ever > > getting the a,b,*c syntax. IOW, the pep may be a step backwards. > > Officially killing that syntax was actually one of the stated goals of the PEP. > > However, the PEP editors would be well within their rights to reject the draft > as 'lacking focus'. I wouldn't be offended given the whole 20 minutes it took to > write it :) By itself, both alternatives presented in the implementation do not "kill" the a,b,*c syntax. I prefer to argue that list.pop(slice) and iunpack(...) do facilitate future support for it, while at the same time removing immediate pressure by providing an easy alternative to support one version of the idiom that still uses current Python syntax. Other potential argument against is that the PEP provides two ways to make it, contrary to the "one way to do it" motto. But both cases, while related, are really focused on solving two different problems; one is a generic solution for iterables (which happend to include a whole bunch of objects, including tuples and lists), and the other is an optimization for mutable lists. There's no reason not to support both (of course, if the whole point of the PEP is accepted). -- Carlos Ribeiro Consultoria em Projetos blog: http://rascunhosrotos.blogspot.com blog: http://pythonnotes.blogspot.com mail: carribeiro@gmail.com mail: carribeiro@yahoo.com From ncoghlan at iinet.net.au Sat Nov 20 19:20:24 2004 From: ncoghlan at iinet.net.au (Nick Coghlan) Date: Sat Nov 20 19:20:33 2004 Subject: [Python-Dev] Re: syntactic shortcut - unpack to variably sizedlist In-Reply-To: <001601c4cf28$7e328ca0$e841fea9@oemcomputer> References: <001601c4cf28$7e328ca0$e841fea9@oemcomputer> Message-ID: <419F8AE8.3010807@iinet.net.au> Raymond Hettinger wrote: >>Officially killing that syntax was actually one of the stated goals of > > the > >>PEP. > > I don't think that is a worthy goal. The is some merit to the a,b,*c > syntax. When the time is ripe, it should have its day in court. There's currently 3 things in the PEP draft: 1. Allow slice objects as arguments list.pop and array.pop 2. Add itertools.iunpack 3. Record the rejection of a, b, *c = L in a PEP (since it's been rejected three times now - once in 2001, and twice this year) The first of those could probably be done without a PEP (given that a strict reading of the documentation suggests that those methods *should* handle slices) The 2nd can certainly be separated out. Since I now know that, for actual iterators (rather than iterables), islice(n) will usually consume the first n elements, it loses some of its appeal (i.e. islice already has most of the features I like about iunpack). The 3rd - well, I thought Guido's rejection in August suggested his position was unlikely to change (the AST branch going in for 2.5 doesn't seem to be significant enough to prompt a change of heart, and I can't think of anything else likely to affect his stated reasons for rejection). I have no great problem with leaving the a, b, *c = L idea unPEPed if there is interest in leaving the idea 'open', though. Cheers, Nick. -- Nick Coghlan | Brisbane, Australia Email: ncoghlan@email.com | Mobile: +61 409 573 268 From ad at papyrus-gmbh.de Sat Nov 20 21:42:53 2004 From: ad at papyrus-gmbh.de (Andreas Degert) Date: Sat Nov 20 21:42:55 2004 Subject: [Python-Dev] Bug in PyLocale_strcoll Message-ID: <87pt289rqq.fsf@pluto.noname> Hello, I think I found a bug in PyLocale_strcoll() (Python 2.3.4). When used with 2 unicode strings, it converts them to wchar strings and uses wcscoll. The bug is that the wchar strings are not 0-terminated. Checking with assert(ws1[len1-1] == 0 && ws2[len2-1] == 0); right before the line result = PyInt_FromLong(wcscoll(ws1, ws2)); confirms the bug. I'm not quite sure what the best fix is. PyUnicode_AsWideChar() copies the unicode chars, but not the terminating 0-char of the unicode string (which is not used in python, but its there anyhow, if I understand the implementation correctly). So one fix would be to change PyUnicode_AsWideChar to copy the terminating 0-char if there's enough space in the output buffer. Another fix would be to terminate the strings in PyLocale_strcoll() before using them: ---------------------------------------------------------- --- _localemodule.c~ Sat Nov 20 21:33:17 2004 +++ _localemodule.c Sat Nov 20 21:35:04 2004 @@ -353,15 +353,19 @@ PyErr_NoMemory(); goto done; } - if (PyUnicode_AsWideChar((PyUnicodeObject*)os1, ws1, len1) == -1) + len1 = PyUnicode_AsWideChar((PyUnicodeObject*)os1, ws1, len1); + if (len1 == -1) goto done; + ws1[len1-1] = 0; ws2 = PyMem_MALLOC(len2 * sizeof(wchar_t)); if (!ws2) { PyErr_NoMemory(); goto done; } - if (PyUnicode_AsWideChar((PyUnicodeObject*)os2, ws2, len2) == -1) + len2 = PyUnicode_AsWideChar((PyUnicodeObject*)os2, ws2, len2); + if (len2 == -1) goto done; + ws2[len2-1] = 0; /* Collate the strings. */ result = PyInt_FromLong(wcscoll(ws1, ws2)); done: ---------------------------------------------------------- cheers Andreas From ad at papyrus-gmbh.de Sat Nov 20 22:05:59 2004 From: ad at papyrus-gmbh.de (Andreas Degert) Date: Sat Nov 20 22:06:01 2004 Subject: [Python-Dev] Bug in PyLocale_strcoll In-Reply-To: <87pt289rqq.fsf@pluto.noname> (Andreas Degert's message of "Sat, 20 Nov 2004 21:42:53 +0100") References: <87pt289rqq.fsf@pluto.noname> Message-ID: <87hdnk9qo8.fsf@pluto.noname> sorry to reply to my own post.. corrected patch following --- _localemodule.c-2.49 Sat Nov 20 22:02:11 2004 +++ _localemodule.c Sat Nov 20 22:01:48 2004 @@ -313,6 +313,7 @@ } if (PyUnicode_AsWideChar((PyUnicodeObject*)os1, ws1, len1) == -1) goto done; + ws1[len1-1] = 0; ws2 = PyMem_MALLOC(len2 * sizeof(wchar_t)); if (!ws2) { PyErr_NoMemory(); @@ -320,6 +321,7 @@ } if (PyUnicode_AsWideChar((PyUnicodeObject*)os2, ws2, len2) == -1) goto done; + ws2[len2-1] = 0; /* Collate the strings. */ result = PyInt_FromLong(wcscoll(ws1, ws2)); done: From aahz at pythoncraft.com Sun Nov 21 00:07:12 2004 From: aahz at pythoncraft.com (Aahz) Date: Sun Nov 21 00:07:15 2004 Subject: [Python-Dev] Bug in PyLocale_strcoll In-Reply-To: <87pt289rqq.fsf@pluto.noname> References: <87pt289rqq.fsf@pluto.noname> Message-ID: <20041120230712.GA801@panix.com> On Sat, Nov 20, 2004, Andreas Degert wrote: > > I think I found a bug in PyLocale_strcoll() (Python 2.3.4). When used > with 2 unicode strings, it converts them to wchar strings and uses > wcscoll. The bug is that the wchar strings are not 0-terminated. If you're sure this is a bug, please file on SF and report back the ID. (If you're not sure, what until you get confirmation from one of the Unicode experts and then file the bug. ;-) -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ WiFi is the SCSI of the 21st Century -- there are fundamental technical reasons for sacrificing a goat. (with no apologies to John Woods) From ncoghlan at iinet.net.au Sun Nov 21 04:43:38 2004 From: ncoghlan at iinet.net.au (Nick Coghlan) Date: Sun Nov 21 04:43:44 2004 Subject: [Python-Dev] Re: syntactic shortcut - unpack to variably sizedlist In-Reply-To: <000001c4cf33$b5a83d00$e841fea9@oemcomputer> References: <000001c4cf33$b5a83d00$e841fea9@oemcomputer> Message-ID: <41A00EEA.4030704@iinet.net.au> Raymond Hettinger wrote: >>I have no great problem with leaving the a, b, *c = L idea unPEPed if >>there is >>interest in leaving the idea 'open', though. > > > Yes. I would like to have it left open. OK, here's what I'll do in the next draft of the proposed PEP (PEPP?): 1. Add a discussion of the advantages of iunpack over islice: - guaranteed minimum length (handy for use with tuple unpacking) - return the iterator as the final object (this is nicer when the original object is an iterable, rather than an iterator - e.g. a list or tuple) 2. Summarise the use cases as given by the previous proponents of the "a, b, *c = S" syntax. 3. Soften the PEP's language regarding the "a, b *c = S" syntax. Point out that this is a way to get the functionality in a fairly easy to spell form for 2.5 that doesn't require changing Python's syntax. itertools.iunpack could serve as a reference for defining the behaviour of that syntax at a later date. I find the following to be rather readable: a, b, itr = iunpack(S, 2) "iunpack" is also easy to look up in the documentation index. If Carlos is amenable, I'll add him as a co-author on the PEP - after all, he's the one who sold *me* on the idea of iunpack, and most of the points defending it are originally his. Cheers, Nick. -- Nick Coghlan | Brisbane, Australia Email: ncoghlan@email.com | Mobile: +61 409 573 268 From skip at pobox.com Sun Nov 21 05:49:20 2004 From: skip at pobox.com (Skip Montanaro) Date: Sun Nov 21 05:47:08 2004 Subject: [Python-Dev] enhanced gdbinit file - looking for Python+Emacs+gdb users Message-ID: <16800.7760.739724.695902@montanaro.dyndns.org> I got a little carried away with the gdbinit file that lives in the Python distribution at Misc/gdbinit today. I decided there's no particular reason that gdb shouldn't display the current Python file when moving up and down the C stack any time it encounters PyEval_EvalFrame. After a little digging I realized this would be pretty easy. I made two changes to the gdbinit file: 1. I wrote a lineno command for gdb that calculates and prints the current Python line number. (It's PyCode_Addr2Line written in gdb's command language.) 2. Using that and the current file gleaned from PyEval_EvalFrame I emit the file:line info in the correct format for Emacs to display the Python line number at the right time using overridden up and down commands. There's one slight hitch. It works perfectly when travelling up the C stack using the up command, however in my environment at least (MacOSX w/ Apple's gcc and gdb) it causes gdb to segfault when travelling down the stack. I can't see anything that would make the behavior different going up instead of down. I need some people to try this thing out to see if it barfs for them as well. Maybe it's just something peculiar to my environment. OTOH, perhaps it's a bug in gdb. Since I can't really check this beast in given its current rather dicey behavior, I dropped a copy at http://www.musi-cal.com/~skip/python/gdbinit If you use Python with gdb via Emacs please give it a whirl and let me know how it works for you. Thanks, Skip P.S. I think I know how to make the gdb print command print Python objects as well... From martin at v.loewis.de Sun Nov 21 10:03:51 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun Nov 21 10:03:58 2004 Subject: [Python-Dev] enhanced gdbinit file - looking for Python+Emacs+gdb users In-Reply-To: <16800.7760.739724.695902@montanaro.dyndns.org> References: <16800.7760.739724.695902@montanaro.dyndns.org> Message-ID: <41A059F7.4080909@v.loewis.de> Skip Montanaro wrote: > There's one slight hitch. It works perfectly when travelling up the C stack > using the up command, however in my environment at least (MacOSX w/ Apple's > gcc and gdb) it causes gdb to segfault when travelling down the stack. I > can't see anything that would make the behavior different going up instead > of down. I need some people to try this thing out to see if it barfs for > them as well. Maybe it's just something peculiar to my environment. OTOH, > perhaps it's a bug in gdb. Without actually looking at your code: a segfault in gdb is always a bug, even if triggered through gdb macros. Regards, Martin From mal at egenix.com Sun Nov 21 15:32:30 2004 From: mal at egenix.com (M.-A. Lemburg) Date: Sun Nov 21 15:32:31 2004 Subject: [Python-Dev] Bug in PyLocale_strcoll In-Reply-To: <20041120230712.GA801@panix.com> References: <87pt289rqq.fsf@pluto.noname> <20041120230712.GA801@panix.com> Message-ID: <41A0A6FE.10203@egenix.com> Aahz wrote: > On Sat, Nov 20, 2004, Andreas Degert wrote: > >>I think I found a bug in PyLocale_strcoll() (Python 2.3.4). When used >>with 2 unicode strings, it converts them to wchar strings and uses >>wcscoll. The bug is that the wchar strings are not 0-terminated. > > > If you're sure this is a bug, please file on SF and report back the ID. > (If you're not sure, what until you get confirmation from one of the > Unicode experts and then file the bug. ;-) Please also check that the bug is still present in Python 2.4 and/or CVS. We've corrected a bug in the PyUnicode_*WideChar*() APIs just recently for Python 2.4. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Nov 21 2004) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From ad at papyrus-gmbh.de Sun Nov 21 23:22:02 2004 From: ad at papyrus-gmbh.de (Andreas Degert) Date: Sun Nov 21 23:22:04 2004 Subject: [Python-Dev] Bug in PyLocale_strcoll In-Reply-To: <41A0A6FE.10203@egenix.com> (M.'s message of "Sun, 21 Nov 2004 15:32:30 +0100") References: <87pt289rqq.fsf@pluto.noname> <20041120230712.GA801@panix.com> <41A0A6FE.10203@egenix.com> Message-ID: <871xemalmd.fsf@pluto.noname> "M.-A. Lemburg" <mal@egenix.com> writes: > Aahz wrote: >> On Sat, Nov 20, 2004, Andreas Degert wrote: >> >>>I think I found a bug in PyLocale_strcoll() (Python 2.3.4). When used >>>with 2 unicode strings, it converts them to wchar strings and uses >>>wcscoll. The bug is that the wchar strings are not 0-terminated. >> If you're sure this is a bug, please file on SF and report back the >> ID. >> (If you're not sure, what until you get confirmation from one of the >> Unicode experts and then file the bug. ;-) > > Please also check that the bug is still present in Python 2.4 and/or > CVS. We've corrected a bug in the PyUnicode_*WideChar*() APIs just > recently for Python 2.4. The off-by-one error fix in unicodeobject.c (2.228 -> 2.229) is correcting a buffer overflow, is just in the same piece of code. I didn't find a clear statement if the unicode string should be 0-terminated or not. In _PyUnicode_New it's 0-terminated, even in the case when it had to call unicode_resize (though there is a comment in unicode_resize "Ux0000 terminated -- XXX is this needed ?"). If these is the only place where unicode objects are created or modified, they seem to be always 0-terminated. wchar strings must be 0-terminated if they are to be used with the wcs* functions. So it's not a good idea to return a non-terminated string from PyUnicode_AsWideChar. If the unicode strings are always 0-terminated (the unicode buffer size is length+1), then we could just change if (size > PyUnicode_GET_SIZE(unicode)) size = PyUnicode_GET_SIZE(unicode); to if (size > PyUnicode_GET_SIZE(unicode)+1) size = PyUnicode_GET_SIZE(unicode)+1; in PyUnicode_AsWideChar to get 0-terminated wchars. Ok... I'm still not sure if I should file a bug for PyLocale_strcoll or PyUnicode_AsWideChar and if the patch for the latter should assume that the unicode string buffer is 0-terminated... cheers Andreas From fdrake at acm.org Mon Nov 22 05:58:38 2004 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Mon Nov 22 05:58:49 2004 Subject: [Python-Dev] TRUNK UNFROZEN In-Reply-To: <200411192203.52490.anthony@interlink.com.au> References: <200411192203.52490.anthony@interlink.com.au> Message-ID: <200411212358.38022.fdrake@acm.org> On Friday 19 November 2004 06:03 am, Anthony Baxter wrote: > Thanks to Tim's latest discovery, it looks like we might need an rc2 - > I might try to aim for this in a week's time (assuming that's ok with > Fred and Martin) and try to keep things moving along to 2.4 final. Works for me. > After which, I'm going to go out and get very drunk <wink> Might wanna join you for that as well. ;-) -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> From oliver.walczak at momatec.de Mon Nov 22 08:29:17 2004 From: oliver.walczak at momatec.de (Oliver Walczak) Date: Mon Nov 22 08:29:19 2004 Subject: [Python-Dev] Constructor bug Message-ID: <FB26DAD89E43D411B25F0050DA517F382E4657@MOMATEC01> Dear List, recently i got stuck on a strange class initialization bug. Please refer the sample attached for a demonstration and watch what happens to d2 in the second class instance. This only works on dictionaries. Any other types initialized outside of __init__ seem to work perfectly. So, is it a bug or a feature? In my eyes this behaviour is not what the programmer has got to expect. Who knows whats happening there and can explain me if it makes any sense? Best regards Oliver -------------- next part -------------- #!/usr/bin/python # -*- coding: iso-8859-1 -*- class test: d1 = {'key':'value'} def __init__(self): self.d2 = {'key':'value'} def Show(self): print 'd1=',self.d1 print 'd2=',self.d2 c1 = test() print 'c1, before:' c1.Show() c1.d1['key'] = 'value2' c1.d2['key'] = 'value2' print 'c1, after:' c1.Show() c2 = test() print 'c2:' c2.Show() From mal at egenix.com Mon Nov 22 09:17:44 2004 From: mal at egenix.com (M.-A. Lemburg) Date: Mon Nov 22 09:17:40 2004 Subject: [Python-Dev] Bug in PyLocale_strcoll In-Reply-To: <871xemalmd.fsf@pluto.noname> References: <87pt289rqq.fsf@pluto.noname> <20041120230712.GA801@panix.com> <41A0A6FE.10203@egenix.com> <871xemalmd.fsf@pluto.noname> Message-ID: <41A1A0A8.8010803@egenix.com> Andreas Degert wrote: > "M.-A. Lemburg" <mal@egenix.com> writes: > > >>Aahz wrote: >> >>>On Sat, Nov 20, 2004, Andreas Degert wrote: >>> >>> >>>>I think I found a bug in PyLocale_strcoll() (Python 2.3.4). When used >>>>with 2 unicode strings, it converts them to wchar strings and uses >>>>wcscoll. The bug is that the wchar strings are not 0-terminated. >>> >>>If you're sure this is a bug, please file on SF and report back the >>>ID. >>>(If you're not sure, what until you get confirmation from one of the >>>Unicode experts and then file the bug. ;-) >> >>Please also check that the bug is still present in Python 2.4 and/or >>CVS. We've corrected a bug in the PyUnicode_*WideChar*() APIs just >>recently for Python 2.4. > > > The off-by-one error fix in unicodeobject.c (2.228 -> 2.229) is > correcting a buffer overflow, is just in the same piece of code. > > I didn't find a clear statement if the unicode string should be > 0-terminated or not. You're right: they are always 0-terminated just like 8-bit strings and even though it doesn't seem to be necessary since Python functions will always use the size field when working on a Unicode object rather than rely on the 0-termination. > In _PyUnicode_New it's 0-terminated, even in the > case when it had to call unicode_resize (though there is a comment in > unicode_resize "Ux0000 terminated -- XXX is this needed ?"). If these > is the only place where unicode objects are created or modified, they > seem to be always 0-terminated. Right. > wchar strings must be 0-terminated if they are to be used with the > wcs* functions. So it's not a good idea to return a non-terminated > string from PyUnicode_AsWideChar. If the unicode strings are always > 0-terminated (the unicode buffer size is length+1), then we could just > change > > if (size > PyUnicode_GET_SIZE(unicode)) > size = PyUnicode_GET_SIZE(unicode); > > to > > if (size > PyUnicode_GET_SIZE(unicode)+1) > size = PyUnicode_GET_SIZE(unicode)+1; > > in PyUnicode_AsWideChar to get 0-terminated wchars. > > Ok... I'm still not sure if I should file a bug for PyLocale_strcoll > or PyUnicode_AsWideChar and if the patch for the latter should assume > that the unicode string buffer is 0-terminated... I think it's probably wise to fix both: Looking again, the patch we applied to PyUnicode_AsWideChar() only fixes the 0-termination problem in the case where HAVE_USABLE_WCHAR_T is set. This should be extended to the memcpy() as well. Still, if the buffer passed to PyUnicode_AsWideChar() is not big enough, you won't get the 0-termination (due to truncation), so PyLocale_strcoll() must be either very careful to allocate a buffer that is always big enough or apply 0-termination itself. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Nov 22 2004) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From ncoghlan at iinet.net.au Mon Nov 22 10:18:20 2004 From: ncoghlan at iinet.net.au (Nick Coghlan) Date: Mon Nov 22 10:18:31 2004 Subject: [Python-Dev] Constructor bug In-Reply-To: <FB26DAD89E43D411B25F0050DA517F382E4657@MOMATEC01> References: <FB26DAD89E43D411B25F0050DA517F382E4657@MOMATEC01> Message-ID: <41A1AEDC.3070105@iinet.net.au> Oliver Walczak wrote: > So, is it a bug or a feature? In my eyes this behaviour is not what the > programmer has got to expect. Who knows whats happening there and can > explain me if it makes any sense? The output is exactly what I expected: c1, before: d1= {'key': 'value'} d2= {'key': 'value'} c1, after: d1= {'key': 'value2'} d2= {'key': 'value2'} c2: d1= {'key': 'value2'} d2= {'key': 'value'} Exactly what do you believe is mysterious here? c2 sees the modified version of the class variable (d1), and sees a fresh copy of the instance variable (d2). Please make you question more explicit. Regards, Nick. -- Nick Coghlan | Brisbane, Australia Email: ncoghlan@email.com | Mobile: +61 409 573 268 From ncoghlan at iinet.net.au Mon Nov 22 10:20:02 2004 From: ncoghlan at iinet.net.au (Nick Coghlan) Date: Mon Nov 22 10:20:06 2004 Subject: [Python-Dev] TRUNK UNFROZEN In-Reply-To: <200411212358.38022.fdrake@acm.org> References: <200411192203.52490.anthony@interlink.com.au> <200411212358.38022.fdrake@acm.org> Message-ID: <41A1AF42.2090908@iinet.net.au> Fred L. Drake, Jr. wrote: > > After which, I'm going to go out and get very drunk <wink> > > Might wanna join you for that as well. ;-) That's a long trip just for a drink! Cheers, Nick. -- Nick Coghlan | Brisbane, Australia Email: ncoghlan@email.com | Mobile: +61 409 573 268 From ncoghlan at iinet.net.au Mon Nov 22 11:36:11 2004 From: ncoghlan at iinet.net.au (Nick Coghlan) Date: Mon Nov 22 11:36:18 2004 Subject: [Python-Dev] Re: syntactic shortcut - unpack to variably sizedlist In-Reply-To: <004e01c4cf83$0166db80$e841fea9@oemcomputer> References: <004e01c4cf83$0166db80$e841fea9@oemcomputer> Message-ID: <41A1C11B.4020808@iinet.net.au> Raymond Hettinger wrote: > Also, if the goal is just to get iunpack(), the good news is that PEP is > not required. Only the a,b,*c syntax would need a PEP. The bad news is > that I have the final say over what goes into itertools (I've rejected > eight proposals so far, including one from Guido). Actually, the goal (or my goal, anyway) was to avoid having the "a, b, *c = seq" idea proposed again in a few months time. Google searches of the py-dev archive can work sometimes, but a PEP is generally more visible. However, if you follow through the rambling below, you'll find it doesn't leave much for a PEP to say. . . so, unless Carlos or anyone else wants to run with it, consider my PEPP from this thread to be dead :) Cheers, Nick. <Rambling summary of the discussion so far> <I included it, since it is what lead me to the second paragraph above> The basic idea of the "a, b, *c = seq" syntax is to unpack the first few items of a sequence into specific variables, and then drop the rest of the sequence into another variable (sometimes the original variable). To me, the presented use cases are nowhere near significant enough to justify special syntax. The other argument in favour (symmetry with extended function call syntax) isn't compelling either, since function calls and assignment statements are, well, different. However, while I didn't find the use cases presented enough to justify a syntax change, I wondered if they pointed towards some potentially missing features. The current PEP draft grew out of the question "Is there a way to provide the _functionality_ without altering Python's syntax?". For objects that support slicing, the following works now: a, b, the_rest = seq[:2] + [seq[2:]] For iterators, roughly equivalent spelling is: a, b, the_rest = [islice(itr, 2)] + [itr] Carlos's suggestion gives a common way to spell it for any iterable: a, b, the_rest = iunpack(iterable, 2) Alternatively, if we stick with different spellings for mutable sequences and iterators, we can get to: a, b = seq.pop(slice(2)) a, b = islice(itr, 2) In both of these cases, the original object (seq, itr) ends up containing "the rest". Neither approach works for an immutable sequence, though - for those, it is still necessary to use the explicit slicing spelling above (or create an iterator and use the islice approach). Anyway, if you're firmly opposed to itertools.iunpack, there isn't much point in pursuing it (since, in the end, your opinion is the one that really counts). Carlos may choose to post it as a recipe over at ASPN. That still leaves adding slice handling to list.pop and array.pop. If nobody else steps forward to do it, I expect I'll eventually get around to coming up with a patch for that. -- Nick Coghlan | Brisbane, Australia Email: ncoghlan@email.com | Mobile: +61 409 573 268 From ad at papyrus-gmbh.de Mon Nov 22 12:46:01 2004 From: ad at papyrus-gmbh.de (Andreas Degert) Date: Mon Nov 22 12:46:04 2004 Subject: [Python-Dev] Bug in PyLocale_strcoll In-Reply-To: <41A1A0A8.8010803@egenix.com> (M.'s message of "Mon, 22 Nov 2004 09:17:44 +0100") References: <87pt289rqq.fsf@pluto.noname> <20041120230712.GA801@panix.com> <41A0A6FE.10203@egenix.com> <871xemalmd.fsf@pluto.noname> <41A1A0A8.8010803@egenix.com> Message-ID: <87sm7285ty.fsf@pluto.noname> "M.-A. Lemburg" <mal@egenix.com> writes: > You're right: they are always 0-terminated just like 8-bit strings > and even though it doesn't seem to be necessary since Python > functions will always use the size field when working on > a Unicode object rather than rely on the 0-termination. OK, should be documented in the code >> Ok... I'm still not sure if I should file a bug for PyLocale_strcoll >> or PyUnicode_AsWideChar and if the patch for the latter should assume >> that the unicode string buffer is 0-terminated... > > I think it's probably wise to fix both: > > Looking again, the patch we applied to PyUnicode_AsWideChar() > only fixes the 0-termination problem in the case where > HAVE_USABLE_WCHAR_T is set. This should be extended to > the memcpy() as well. What I read from the code is that now in both cases the string is copied without 0 and that is consistent with the size the buffer is checked for (PyUnicode_GET_SIZE gives the value of the length field and that doesn't include the 0-termination) > Still, if the buffer passed to PyUnicode_AsWideChar() > is not big enough, you won't get the 0-termination (due > to truncation), so PyLocale_strcoll() must be either very > careful to allocate a buffer that is always big enough > or apply 0-termination itself. PyLocale_strcoll() acts quite careful but even so it didn't get what it expected ;-). This bug is masked by the bug you referred to when the copy loop is used (ie. if wchar sizes don't match) and the output buffer string is big enough (like in the strcoll case because the buffer size already accounts for the 0-termination). I appended a (untested) patch for unicodeobject.c. The documentation should be clarified too. Would a patch against concrete.tex be accepted where I change - 'Unicode object' to 'Unicode string' when only the string part of the python object is referenced, - 'size of the object' to 'length of the string' - mention the 0-termination of the return-value of PyUnicode_AS_UNICODE() - mention the 0-termination of the return-value of PyUnicode_AsWideChar - '... represents a 16-bit...' to something that explains 16 vs. 32 but depending on internal representation (UCS-2 or UCS-4) selected at compile time --- unicodeobject.c-2.229 Mon Nov 22 10:58:42 2004 +++ unicodeobject.c Mon Nov 22 11:10:07 2004 @@ -144,8 +144,7 @@ return -1; } - /* We allocate one more byte to make sure the string is - Ux0000 terminated -- XXX is this needed ? */ + /* We allocate one more Py_UNICODE and make the string Ux0000 terminated */ oldstr = unicode->str; PyMem_RESIZE(unicode->str, Py_UNICODE, length + 1); if (!unicode->str) { @@ -167,8 +166,7 @@ return 0; } -/* We allocate one more byte to make sure the string is - Ux0000 terminated -- XXX is this needed ? +/* We allocate length+1 and make the string Ux0000 terminated XXX This allocator could further be enhanced by assuring that the free list never reduces its size below 1. @@ -384,8 +382,10 @@ PyErr_BadInternalCall(); return -1; } - if (size > PyUnicode_GET_SIZE(unicode)) - size = PyUnicode_GET_SIZE(unicode); + /* copy all chars including 0-termination if the output buffer + size is sufficient */ + if (size > PyUnicode_GET_SIZE(unicode)+1) + size = PyUnicode_GET_SIZE(unicode)+1; #ifdef HAVE_USABLE_WCHAR_T memcpy(w, unicode->str, size * sizeof(wchar_t)); #else From ncoghlan at iinet.net.au Mon Nov 22 12:52:24 2004 From: ncoghlan at iinet.net.au (Nick Coghlan) Date: Mon Nov 22 12:52:28 2004 Subject: [Python-Dev] Current CVS, Cygwin and "make test" In-Reply-To: <419F7535.9050802@iinet.net.au> References: <419E0B4E.7040109@iinet.net.au> <20041119232733.GB1680@tishler.net> <419EABE2.9040804@iinet.net.au> <20041120150637.GA3024@tishler.net> <419F7535.9050802@iinet.net.au> Message-ID: <41A1D2F8.9040900@iinet.net.au> Nick Coghlan wrote: > Anyway, I won't be posting any more about this on py-dev, since I've > established that *my* problems probably aren't Python related. I lied. The *real* final word - some digging with Google shows that there are still some issues with Cygwin and hyperthreaded processors. Sure enough, my desktop is a hyper-threaded P4. Just a minor detail the "Known Problems" section in the Cygwin FAQ neglects to mention. . . Cheers, Nick. -- Nick Coghlan | Brisbane, Australia Email: ncoghlan@email.com | Mobile: +61 409 573 268 From oliver.walczak at momatec.de Mon Nov 22 13:20:00 2004 From: oliver.walczak at momatec.de (Oliver Walczak) Date: Mon Nov 22 13:20:01 2004 Subject: AW: [Python-Dev] Constructor bug In-Reply-To: <FB26DAD89E43D411B25F0050DA517F38365308@MOMATEC01> Message-ID: <FB26DAD89E43D411B25F0050DA517F382E4659@MOMATEC01> Dear Nick, dear Andreas, Imagine you add two ints i1 and i2, one in and one outside of __init__, none of them gets shared to other instances of test. They are always initialized to what the class definition says to. Please refer the modified example. In your sense, Andreas, c2 should also see a modified version of i1. And in yours, Nick, i1 should be a unique int shared by all instances of test. That's not true actually. In my opinion, initialization outside of __init__ without self. and inside __init__ with self. should work absolutely identical. Best regards Oliver Oliver Walczak wrote: > So, is it a bug or a feature? In my eyes this behaviour is not what the > programmer has got to expect. Who knows whats happening there and can > explain me if it makes any sense? The output is exactly what I expected: c1, before: d1= {'key': 'value'} d2= {'key': 'value'} c1, after: d1= {'key': 'value2'} d2= {'key': 'value2'} c2: d1= {'key': 'value2'} d2= {'key': 'value'} Exactly what do you believe is mysterious here? c2 sees the modified version of the class variable (d1), and sees a fresh copy of the instance variable (d2). Please make you question more explicit. Regards, Nick. "Oliver Walczak" <oliver.walczak@momatec.de> writes: > Dear List, > > recently i got stuck on a strange class initialization bug. Please refer the > sample attached for a demonstration and watch what happens to d2 in the > second class instance. > This only works on dictionaries. Any other types initialized outside of > __init__ seem to work perfectly. Did you try with type list? You modify the object referenced by d1, and that object is created when the class is defined. All instances of test share the dictionary referenced by d2, and have a unique dictionary referenced by d1. That what your class definition says, so I'd say it's a feature not a bug. > > So, is it a bug or a feature? In my eyes this behaviour is not what the > programmer has got to expect. Who knows whats happening there and can > explain me if it makes any sense? > > Best regards > Oliver > > #!/usr/bin/python > # -*- coding: iso-8859-1 -*- > > class test: > d1 = {'key':'value'} > def __init__(self): > self.d2 = {'key':'value'} > > def Show(self): > print 'd1=',self.d1 > print 'd2=',self.d2 > > c1 = test() > print 'c1, before:' > c1.Show() > c1.d1['key'] = 'value2' > c1.d2['key'] = 'value2' > print 'c1, after:' > c1.Show() > > c2 = test() > print 'c2:' > c2.Show() -------------- next part -------------- #!/usr/bin/python # -*- coding: iso-8859-1 -*- class test: d1 = {'key':'value'} i1 = 1 def __init__(self): self.d2 = {'key':'value'} self.i2 = 1 def Show(self): from pprint import pprint print 'd1=', pprint(self.d1) print 'd2=', pprint(self.d2) print 'i1=',self.i1 print 'i2=',self.i2 c1 = test() print 'c1, before:' c1.Show() c1.d1['key'] = 'value2' c1.d2['key'] = 'value2' c1.i1 = 2 c1.i2 = 2 print 'c1, after:' c1.Show() c2 = test() print 'c2:' c2.Show() From fdrake at acm.org Mon Nov 22 13:21:42 2004 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Mon Nov 22 13:21:57 2004 Subject: [Python-Dev] TRUNK UNFROZEN In-Reply-To: <41A1AF42.2090908@iinet.net.au> References: <200411192203.52490.anthony@interlink.com.au> <200411212358.38022.fdrake@acm.org> <41A1AF42.2090908@iinet.net.au> Message-ID: <200411220721.42132.fdrake@acm.org> On Monday 22 November 2004 04:20 am, Nick Coghlan wrote: > That's a long trip just for a drink! Yeah, well, it's unusual enough that I have more than one or two beers that it seems somehow... fitting. Of course, it may be a trip made "in spirit." ;-) -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> From ncoghlan at iinet.net.au Mon Nov 22 13:39:47 2004 From: ncoghlan at iinet.net.au (Nick Coghlan) Date: Mon Nov 22 13:39:53 2004 Subject: AW: [Python-Dev] Constructor bug In-Reply-To: <FB26DAD89E43D411B25F0050DA517F382E4659@MOMATEC01> References: <FB26DAD89E43D411B25F0050DA517F382E4659@MOMATEC01> Message-ID: <41A1DE13.7090402@iinet.net.au> Short answer for py-dev: This is not a bug. It is how the language works. I'll give some further explanation in private mail - any additional queries should go to comp.lang.python. Cheers, Nick. -- Nick Coghlan | Brisbane, Australia Email: ncoghlan@email.com | Mobile: +61 409 573 268 From mal at egenix.com Mon Nov 22 14:03:23 2004 From: mal at egenix.com (M.-A. Lemburg) Date: Mon Nov 22 14:03:29 2004 Subject: [Python-Dev] Bug in PyLocale_strcoll In-Reply-To: <87sm7285ty.fsf@pluto.noname> References: <87pt289rqq.fsf@pluto.noname> <20041120230712.GA801@panix.com> <41A0A6FE.10203@egenix.com> <871xemalmd.fsf@pluto.noname> <41A1A0A8.8010803@egenix.com> <87sm7285ty.fsf@pluto.noname> Message-ID: <41A1E39B.3030308@egenix.com> Andreas Degert wrote: > "M.-A. Lemburg" <mal@egenix.com> writes: > > >>You're right: they are always 0-terminated just like 8-bit strings >>and even though it doesn't seem to be necessary since Python >>functions will always use the size field when working on >>a Unicode object rather than rely on the 0-termination. > > > OK, should be documented in the code It is, but I wasn't sure whether it is really such a good idea to waist the extra memory and wanted to keep the option of removing the 0-termination. >>>Ok... I'm still not sure if I should file a bug for PyLocale_strcoll >>>or PyUnicode_AsWideChar and if the patch for the latter should assume >>>that the unicode string buffer is 0-terminated... >> >>I think it's probably wise to fix both: >> >>Looking again, the patch we applied to PyUnicode_AsWideChar() >>only fixes the 0-termination problem in the case where >>HAVE_USABLE_WCHAR_T is set. This should be extended to >>the memcpy() as well. > > > What I read from the code is that now in both cases the string is > copied without 0 and that is consistent with the size the buffer is > checked for (PyUnicode_GET_SIZE gives the value of the length field > and that doesn't include the 0-termination) > > >>Still, if the buffer passed to PyUnicode_AsWideChar() >>is not big enough, you won't get the 0-termination (due >>to truncation), so PyLocale_strcoll() must be either very >>careful to allocate a buffer that is always big enough >>or apply 0-termination itself. > > > PyLocale_strcoll() acts quite careful but even so it didn't get what > it expected ;-). This bug is masked by the bug you referred to when > the copy loop is used (ie. if wchar sizes don't match) and the output > buffer string is big enough (like in the strcoll case because the > buffer size already accounts for the 0-termination). > > I appended a (untested) patch for unicodeobject.c. I've just checked in a patch which should correct the problem. > The documentation should be clarified too. Would a patch against > concrete.tex be accepted where I change > > - 'Unicode object' to 'Unicode string' when only the string part of > the python object is referenced, Not sure what you mean here. > - 'size of the object' to 'length of the string' Dito. > - mention the 0-termination of the return-value of > PyUnicode_AS_UNICODE() > > - mention the 0-termination of the return-value of > PyUnicode_AsWideChar I don't think we should document this. Programmers should always use the size of the object rather than rely on the 0-termination. > - '... represents a 16-bit...' to something that explains 16 vs. 32 > but depending on internal representation (UCS-2 or UCS-4) selected at > compile time +1 -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Nov 22 2004) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From theller at python.net Mon Nov 22 15:22:31 2004 From: theller at python.net (Thomas Heller) Date: Mon Nov 22 15:21:42 2004 Subject: [Python-Dev] Bug in PyLocale_strcoll In-Reply-To: <41A1E39B.3030308@egenix.com> (M.'s message of "Mon, 22 Nov 2004 14:03:23 +0100") References: <87pt289rqq.fsf@pluto.noname> <20041120230712.GA801@panix.com> <41A0A6FE.10203@egenix.com> <871xemalmd.fsf@pluto.noname> <41A1A0A8.8010803@egenix.com> <87sm7285ty.fsf@pluto.noname> <41A1E39B.3030308@egenix.com> Message-ID: <hdni2cbc.fsf@python.net> >> - mention the 0-termination of the return-value of >> PyUnicode_AS_UNICODE() >> - mention the 0-termination of the return-value of >> PyUnicode_AsWideChar > > I don't think we should document this. Programmers should always > use the size of the object rather than rely on the 0-termination. Then, *this* should be clearly mentioned somewhere. imo. Thomas From mal at egenix.com Mon Nov 22 15:30:38 2004 From: mal at egenix.com (M.-A. Lemburg) Date: Mon Nov 22 15:30:42 2004 Subject: [Python-Dev] Bug in PyLocale_strcoll In-Reply-To: <hdni2cbc.fsf@python.net> References: <87pt289rqq.fsf@pluto.noname> <20041120230712.GA801@panix.com> <41A0A6FE.10203@egenix.com> <871xemalmd.fsf@pluto.noname> <41A1A0A8.8010803@egenix.com> <87sm7285ty.fsf@pluto.noname> <41A1E39B.3030308@egenix.com> <hdni2cbc.fsf@python.net> Message-ID: <41A1F80E.8000508@egenix.com> Thomas Heller wrote: >>>- mention the 0-termination of the return-value of >>> PyUnicode_AS_UNICODE() >>>- mention the 0-termination of the return-value of >>> PyUnicode_AsWideChar >> >>I don't think we should document this. Programmers should always >>use the size of the object rather than rely on the 0-termination. > > > Then, *this* should be clearly mentioned somewhere. imo. FYI, I added this to the PyUnicode_AsWideChar() documentation. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Nov 22 2004) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From carribeiro at gmail.com Mon Nov 22 15:57:46 2004 From: carribeiro at gmail.com (Carlos Ribeiro) Date: Mon Nov 22 15:58:18 2004 Subject: [Python-Dev] Re: syntactic shortcut - unpack to variably sizedlist In-Reply-To: <41A1C11B.4020808@iinet.net.au> References: <004e01c4cf83$0166db80$e841fea9@oemcomputer> <41A1C11B.4020808@iinet.net.au> Message-ID: <864d37090411220657703083a8@mail.gmail.com> On Mon, 22 Nov 2004 20:36:11 +1000, Nick Coghlan <ncoghlan@iinet.net.au> wrote: > Anyway, if you're firmly opposed to itertools.iunpack, there isn't much point in > pursuing it (since, in the end, your opinion is the one that really counts). > Carlos may choose to post it as a recipe over at ASPN. I also grew some doubts about iunpack(), as currently proposed. I still feel that it has enough of a use case to deserve discussion, but I'm not so sure about the details. I did check itertools before submitting my original implementation, and for some reason, I didn't perceive that islice coud be used for a similar effect with a slightly different usage pattern. One of the advantages of writing a PEP (or pre-PEP, as it is now) is that, even if not absolutely required for the proposed addition to itertools, it's still a way to encourage discussion and document the results. I also still think that the pre-PEP could be published on the main c.l.py list, for discussion and feedback. The key point is the comparison between iunpack and islice. If iunpack is deemed to be just an obvious extension to behavior that can already be achieved with iunpack, then fine -- forget it. The most that I can propose is to include an example in the documentation on how to achieve iunpack behavior with islice. On the other hand, if it becomes clear that the actual usage pattern for iunpack and islice are in fact different, then I would stand for the addition of iunpack. Of course, Raymond has the final word :-) -- Carlos Ribeiro Consultoria em Projetos blog: http://rascunhosrotos.blogspot.com blog: http://pythonnotes.blogspot.com mail: carribeiro@gmail.com mail: carribeiro@yahoo.com From misa at redhat.com Mon Nov 22 19:52:11 2004 From: misa at redhat.com (Mihai Ibanescu) Date: Mon Nov 22 19:52:14 2004 Subject: [Python-Dev] Multilib strikes back Message-ID: <20041122185211.GL19174@abulafia.devel.redhat.com> Hello, This is yet another pain caused by the existance of both 32 and 64-bit libraries/binaries on the same system This is the bug that landed on my plate: http://bugzilla.redhat.com/beta/show_bug.cgi?id=139911 Even though I had no plan to have both 32 and 64-bit python on the same system, the reporter of this bug is trying to compule a 32-bit version of libxml on a 64-bit system, and is getting into weird errors, mostly because the .h files have the wrong definition. Please see comment #2 from Jakub about possible solutions. If one has strong preferences one way or another please let me know and I'll write the patch, otherwise I tend to favor the (int(sizeof()) option. Thanks, Mihai From tim.peters at gmail.com Mon Nov 22 20:04:17 2004 From: tim.peters at gmail.com (Tim Peters) Date: Mon Nov 22 20:04:21 2004 Subject: [Python-Dev] Multilib strikes back In-Reply-To: <20041122185211.GL19174@abulafia.devel.redhat.com> References: <20041122185211.GL19174@abulafia.devel.redhat.com> Message-ID: <1f7befae0411221104119c1a08@mail.gmail.com> [Mihai Ibanescu] > This is yet another pain caused by the existance of both 32 and 64-bit > libraries/binaries on the same system > > This is the bug that landed on my plate: > > http://bugzilla.redhat.com/beta/show_bug.cgi?id=139911 > > Even though I had no plan to have both 32 and 64-bit python on the > same system, the reporter of this bug is trying to compule a 32-bit > version of libxml on a 64-bit system, and is getting into weird errors, > mostly because the .h files have the wrong definition. > > Please see comment #2 from Jakub about possible solutions. If one > has strong preferences one way or another please let me know and I'll > write the patch, otherwise I tend to favor the (int(sizeof()) option. I can't make time to pretend to understand this, but this solution won't fly: Particularly for the SIZEOF_* defines, one solution is to change them from constants to sizeof (long), sizeof (double) etc. In C that should make absolutely no difference, ... The SIZEOF_* defines are used in C preprocessor #if expressions, and sizeof() can't be used in that context. That's why Python config defines them as integer literals to begin with. From misa at redhat.com Mon Nov 22 20:49:39 2004 From: misa at redhat.com (Mihai Ibanescu) Date: Mon Nov 22 20:49:48 2004 Subject: [Python-Dev] Multilib strikes back In-Reply-To: <1f7befae0411221104119c1a08@mail.gmail.com> References: <20041122185211.GL19174@abulafia.devel.redhat.com> <1f7befae0411221104119c1a08@mail.gmail.com> Message-ID: <20041122194939.GN19174@abulafia.devel.redhat.com> On Mon, Nov 22, 2004 at 02:04:17PM -0500, Tim Peters wrote: > [Mihai Ibanescu] > > This is yet another pain caused by the existance of both 32 and 64-bit > > libraries/binaries on the same system > > > > This is the bug that landed on my plate: > > > > http://bugzilla.redhat.com/beta/show_bug.cgi?id=139911 > > > > Even though I had no plan to have both 32 and 64-bit python on the > > same system, the reporter of this bug is trying to compule a 32-bit > > version of libxml on a 64-bit system, and is getting into weird errors, > > mostly because the .h files have the wrong definition. > > > > Please see comment #2 from Jakub about possible solutions. If one > > has strong preferences one way or another please let me know and I'll > > write the patch, otherwise I tend to favor the (int(sizeof()) option. > > I can't make time to pretend to understand this, but this solution won't fly: > > Particularly for the SIZEOF_* defines, one solution is to change them > from constants to sizeof (long), sizeof (double) etc. In C that should > make absolutely no difference, ... > > The SIZEOF_* defines are used in C preprocessor #if expressions, and > sizeof() can't be used in that context. That's why Python config > defines them as integer literals to begin with. Indeed, that would be a problem. How about a _pyconfig_32.h and a _pyconfig_64.h and have pyconfig.h include one or the other, depending on the environment? Thanks again! Misa From skip at pobox.com Mon Nov 22 21:01:24 2004 From: skip at pobox.com (Skip Montanaro) Date: Mon Nov 22 20:58:32 2004 Subject: [Python-Dev] Multilib strikes back In-Reply-To: <20041122194939.GN19174@abulafia.devel.redhat.com> References: <20041122185211.GL19174@abulafia.devel.redhat.com> <1f7befae0411221104119c1a08@mail.gmail.com> <20041122194939.GN19174@abulafia.devel.redhat.com> Message-ID: <16802.17812.721194.512642@montanaro.dyndns.org> Mihai> How about a _pyconfig_32.h and a _pyconfig_64.h and have Mihai> pyconfig.h include one or the other, depending on the Mihai> environment? How would that be detected? As I understand the original bug report, the user gave gcc a -m32 flag. How would that be reflected in the environment? Let's assume you can coax libxml into compiling in 32-bit mode with Python support enabled even though you have a 64-bit mode Python installed. Won't that just push the problem further down the tool chain? Instead of a compilation failure wouldn't you get a runtime error of some sort, or would the two coexist? Skip From martin at v.loewis.de Mon Nov 22 23:42:18 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon Nov 22 23:42:15 2004 Subject: [Python-Dev] Multilib strikes back In-Reply-To: <20041122185211.GL19174@abulafia.devel.redhat.com> References: <20041122185211.GL19174@abulafia.devel.redhat.com> Message-ID: <41A26B4A.3080401@v.loewis.de> Mihai Ibanescu wrote: > Please see comment #2 from Jakub about possible solutions. If one has strong > preferences one way or another please let me know and I'll write the patch, > otherwise I tend to favor the (int(sizeof()) option. I like Jakub's third proposal best: have three different pyconfig.h files, and install them into different locations. I assume that, in a multilib installation, there are per-lib directories in the include path - this is where each different pyconfig.h should go. Regards, Martin From tlb at tlb.org Mon Nov 22 23:11:23 2004 From: tlb at tlb.org (Trevor Blackwell) Date: Tue Nov 23 03:02:23 2004 Subject: [Python-Dev] PEP318 and @decorators Message-ID: <1101161483.55773.352.camel@lab> I just read the plan for decorators in Python 2.4, and I'm surprised to see no discussion of what seems to me the cleanest answer, to optionally write function definitions as assignments: foo=def(a,b): print a,b which allows trivial modification by decorators: bar1=staticmethod(def(a, b)): print a,b bar2=accepts(int, int, staticmethod(def(a, b))): print a,b bar3=adddocstring("this function prints its arguments", trace(def(a,b))): print a,b It makes it clear in which order the functions are called, unlike with the @ syntax, and what their arguments are. It has obvious line-break syntax. You can include arguments to the functions without having to curry them, making it much easier to write one's own decorator functions. Both the "def foo(...):" and "foo=def(...):" syntaxes would be accepted. I don't think there's any ambiguity. Classes would work the same way: SimpleClass=class(object): ... FancyClass=tweakclass(class(object)): ... Among other things, by unifying assignment and definition, it makes the semantics of function name (re-)binding obvious, so if you read code like, foo=def(a,b): print "goodbye" foo2=foo foo=def(c,b): print "hello" foo() foo2() it's obvious what will happen, while with the current syntax people familiar with other languages would expect an error or undefined behavior. It also works nicely for nested functions, making it obvious that it binds in the local namespace just like regular assignment. Making function definition just another sort of assignment emphasizes the dynamic nature of the language, and will encourage people to treat functions as first-class data types. The precise rule is that when there is a def or class in a line (and there can be only one) there must be a colon and a block following. You could even use this to provide a more general alternative to lambda, allowing multi-statement lambdas: x=map(def(a), [1,2,3]): b=sin(a) return b*3 assert x == [3*sin(1), 3*sin(2), 3*sin(3)] though this might not be something to encourage. And it makes stashing functions somewhere easy: atexit.register(def()): print "exiting..." converters.append(def(a)): ... convert something ... Etags doesn't currently grok this syntax, but it should anyway because it would also help tag global variables. It's easy to implement with a regex, something like /^\s*(\w+)\s*=.*\bdef\b/ One bummer is that it may be more difficult to set the __name__ property of a function. -- Trevor Blackwell tlb@tlb.org (650) 776-7870 From misa at redhat.com Tue Nov 23 03:19:14 2004 From: misa at redhat.com (Mihai Ibanescu) Date: Tue Nov 23 03:19:20 2004 Subject: [Python-Dev] Multilib strikes back In-Reply-To: <16802.17812.721194.512642@montanaro.dyndns.org> References: <20041122185211.GL19174@abulafia.devel.redhat.com> <1f7befae0411221104119c1a08@mail.gmail.com> <20041122194939.GN19174@abulafia.devel.redhat.com> <16802.17812.721194.512642@montanaro.dyndns.org> Message-ID: <20041123021914.GC26050@abulafia.devel.redhat.com> On Mon, Nov 22, 2004 at 02:01:24PM -0600, Skip Montanaro wrote: > Mihai> How about a _pyconfig_32.h and a _pyconfig_64.h and have > Mihai> pyconfig.h include one or the other, depending on the > Mihai> environment? > > How would that be detected? As I understand the original bug report, the > user gave gcc a -m32 flag. How would that be reflected in the environment? With the help of Jakub: pyconfig.h would have something close to: #include <limits.h> #if CHAR_BIT == 8 && LONG_MAX == 0x7fffffff #include "_pyconfig_32.h" #elif CHAR_BIT == 8 && LONG_MAX == 0x7fffffffffffffff #include "_pyconfig_64.h" #else #error Unable to detect architecture word length #endif This way, if the compiler is in 32-bit mode, the proper file is included. > Let's assume you can coax libxml into compiling in 32-bit mode with Python > support enabled even though you have a 64-bit mode Python installed. Won't > that just push the problem further down the tool chain? Instead of a > compilation failure wouldn't you get a runtime error of some sort, or would > the two coexist? Ideally, and that is something I would like to tackle asap, it should be possible to have both a 32-bit and a 64-bit version of python on the same machine. This is not exactly possible currently since the binaries are named the same, but that's the whole purpose of multilib :-) Misa From kbk at shore.net Tue Nov 23 05:02:32 2004 From: kbk at shore.net (Kurt B. Kaiser) Date: Tue Nov 23 05:02:39 2004 Subject: [Python-Dev] Bugfix to 2.4? Message-ID: <87r7mlnrfr.fsf@hydra.localdomain> Hye-Shik Chang worked out a fix for a thread / interrupt bug. The fix involves ceval.c. www.python.org/sf/875692 Skip and I have reviewed it. IMO it seems to be a minor risk, should I check it in? -- KBK From foom at fuhm.net Tue Nov 23 05:56:46 2004 From: foom at fuhm.net (James Y Knight) Date: Tue Nov 23 05:57:30 2004 Subject: [Python-Dev] Multilib strikes back In-Reply-To: <20041123021914.GC26050@abulafia.devel.redhat.com> References: <20041122185211.GL19174@abulafia.devel.redhat.com> <1f7befae0411221104119c1a08@mail.gmail.com> <20041122194939.GN19174@abulafia.devel.redhat.com> <16802.17812.721194.512642@montanaro.dyndns.org> <20041123021914.GC26050@abulafia.devel.redhat.com> Message-ID: <1400815C-3D0C-11D9-96B0-000A95A50FB2@fuhm.net> On Nov 22, 2004, at 9:19 PM, Mihai Ibanescu wrote: > On Mon, Nov 22, 2004 at 02:01:24PM -0600, Skip Montanaro wrote: >> Mihai> How about a _pyconfig_32.h and a _pyconfig_64.h and have >> Mihai> pyconfig.h include one or the other, depending on the >> Mihai> environment? >> >> How would that be detected? As I understand the original bug report, >> the >> user gave gcc a -m32 flag. How would that be reflected in the >> environment? > > With the help of Jakub: pyconfig.h would have something close to: > > #include <limits.h> > > #if CHAR_BIT == 8 && LONG_MAX == 0x7fffffff > #include "_pyconfig_32.h" > #elif CHAR_BIT == 8 && LONG_MAX == 0x7fffffffffffffff > #include "_pyconfig_64.h" > #else > #error Unable to detect architecture word length > #endif > > This way, if the compiler is in 32-bit mode, the proper file is > included. I think that is the wrong solution. The right solution is to put pyconfig.h in an arch-specific directory. In current standards, that'd probably have to be /usr/lib*: e.g. /usr/lib/python2.4/include/ and /usr/lib64/python2.4/include/. In the proposed multiarch standard (http://www.linuxbase.org/~taggart/multiarch.html), that would be /usr/include/i386-linux/ and /usr/include/x86_64-linux/. Different, but slightly related: since .py files aren't platform dependent, /usr/lib* is really the wrong place for them. They should be going in /usr/share/python2.X/ not /usr/lib*/python2.X/. Obviously the .so files still need to go in /usr/lib*, though, so that might be a bit of a trick. James From python at rcn.com Tue Nov 23 06:23:04 2004 From: python at rcn.com (Raymond Hettinger) Date: Tue Nov 23 06:25:44 2004 Subject: [Python-Dev] Bugfix to 2.4? In-Reply-To: <87r7mlnrfr.fsf@hydra.localdomain> Message-ID: <008901c4d11c$82734b60$e841fea9@oemcomputer> [Kurt B. Kaiser] > > Hye-Shik Chang worked out a fix for a thread / interrupt bug. The fix > involves ceval.c. > > www.python.org/sf/875692 > > Skip and I have reviewed it. IMO it seems to be a minor risk, should > I check it in? It's up to Anthony. I would like to see it go in. Raymond From tjreedy at udel.edu Tue Nov 23 08:32:54 2004 From: tjreedy at udel.edu (Terry Reedy) Date: Tue Nov 23 08:32:45 2004 Subject: [Python-Dev] Re: PEP318 and @decorators References: <1101161483.55773.352.camel@lab> Message-ID: <cnup2p$9nh$1@sea.gmane.org> "Trevor Blackwell" <tlb@tlb.org> wrote in message news:1101161483.55773.352.camel@lab... > > I just read the plan for decorators in Python 2.4, and I'm surprised to > see no discussion You are a bit late. Alternatives to @ were discussed last August/September in hundreds of posts on pydev and c.l.p/mailing list and summarized on the Wiki. The community selected one which was ably presented to Guido who seriously considered it but ultimately rejected it. End of discussion. 2.4 is nearly done and ready for release. > of what seems to me the cleanest answer, to optionally > write function definitions as assignments: The issue of multiline anonymous function definitions is mostly independent of decorators. Requests and informal proposals in this regard have appeared on c.l.p for years and, I believe, been ignored or informally rejected by Guido. But I don't know that there has been a formal PEP to be formally rejected. I suspect that most developers would prefer that further discussion of this issue continue on c.l.p and the corresponding python-list. Terry J. Reedy From jason at tishler.net Tue Nov 23 13:45:53 2004 From: jason at tishler.net (Jason Tishler) Date: Tue Nov 23 13:44:12 2004 Subject: [Python-Dev] Current CVS, Cygwin and "make test" In-Reply-To: <20041120150637.GA3024@tishler.net> References: <419E0B4E.7040109@iinet.net.au> <20041119232733.GB1680@tishler.net> <419EABE2.9040804@iinet.net.au> <20041120150637.GA3024@tishler.net> Message-ID: <20041123124553.GB2096@tishler.net> On Sat, Nov 20, 2004 at 10:06:37AM -0500, Jason Tishler wrote: > > If you (or anyone else) could check that the unit tests work on > > Cygwin for you, that would make me more certain that this is just a > > problem with my own Cygwin setup, and not related to anything in > > 2.4c1 that wasn't in 2.4b1. > > I just upgraded one of my machines to Cygwin 1.5.12, but I did not > upgrade the other packages. I get the following results: > > failures: > > test_shutil (passed under older Cygwin and/or Python versions) > test_subprocess (new test) > test_tcl (new test) > > hangs: > > test_threaded_import (passed under older Cygwin and/or ... > > I will investigate the above and report back. AFAICT, the above (with possibly one exception) are due to Cygwin bugs. I can provide more details if anyone is interested. On Sat, Nov 20, 2004 at 01:03:42AM +1000, Nick Coghlan wrote: > This was meant to be a "Yup, she's all good" exercise before 2.4 final > goes out the door. IMO, "Yup, she's all good!" :,) The test_tcl test case is failing under Cygwin because it expects loadtk() to fail if the DISPLAY environment variable is not set. However, under Cygwin we have the following: >>> import os, Tkinter >>> del os.environ['DISPLAY'] >>> Tkinter.Tcl().loadtk() >>> The above works because Cygwin's Tcl/Tk renders directly on the Windows GDI instead of going through an X Server. Can I submit a patch to test_tcl to skip this test as is done for win and darwin? Thanks, Jason -- PGP/GPG Key: http://www.tishler.net/jason/pubkey.asc or key servers Fingerprint: 7A73 1405 7F2B E669 C19D 8784 1AFD E4CC ECF4 8EF6 From jason at tishler.net Tue Nov 23 13:47:22 2004 From: jason at tishler.net (Jason Tishler) Date: Tue Nov 23 13:45:39 2004 Subject: [Python-Dev] Current CVS, Cygwin and "make test" In-Reply-To: <41A1D2F8.9040900@iinet.net.au> References: <419E0B4E.7040109@iinet.net.au> <20041119232733.GB1680@tishler.net> <419EABE2.9040804@iinet.net.au> <20041120150637.GA3024@tishler.net> <419F7535.9050802@iinet.net.au> <41A1D2F8.9040900@iinet.net.au> Message-ID: <20041123124722.GC2096@tishler.net> Nick, On Mon, Nov 22, 2004 at 09:52:24PM +1000, Nick Coghlan wrote: > Nick Coghlan wrote: > >Anyway, I won't be posting any more about this on py-dev, since I've > >established that *my* problems probably aren't Python related. > > I lied. The *real* final word - some digging with Google shows that > there are still some issues with Cygwin and hyperthreaded processors. > Sure enough, my desktop is a hyper-threaded P4. Mine too. I guess the above may explain why I can't get test_threaded_import to hang again... Jason -- PGP/GPG Key: http://www.tishler.net/jason/pubkey.asc or key servers Fingerprint: 7A73 1405 7F2B E669 C19D 8784 1AFD E4CC ECF4 8EF6 From anthony at interlink.com.au Tue Nov 23 14:19:53 2004 From: anthony at interlink.com.au (Anthony Baxter) Date: Tue Nov 23 14:20:11 2004 Subject: [Python-Dev] py2.4 and gcc4 Message-ID: <200411240019.54226.anthony@interlink.com.au> Fedora rawhide now ships with a gcc4 preview package - I thought I'd build current-cvs with it. Readline fails to build with gcc4 -pthread -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -fPIC -fno-strict-aliasing -I. -I/home/anthony/src/py/pyhead/dist/src/./Include -I/usr/local/include -I/home/anthony/src/py/pyhead/dist/src/Include -I/home/anthony/src/py/pyhead/dist/src -c /home/anthony/src/py/pyhead/dist/src/Modules/readline.c -o build/temp.linux-i686-2.4/readline.o /home/anthony/src/py/pyhead/dist/src/Modules/readline.c:97: error: static declaration of ?history_length? follows non-static declaration /usr/include/readline/history.h:224: error: previous declaration of ?history_length? was here test_cfgparser causes a segfault, unfortunately. Since this doesn't occur with any other compilers that I know of (*) I'm inclined to blame gcc4, and not bother too much with it further. I'm pretty short of time, but if someone wants to spend time figuring out why, let me know and I'll dump a SF bugreport with the stack trace &c... Anthony (*) Ok - the HP compiler on HP/UX on Itanium causes segfaults and random breakages _everywhere_ but that's because of the platform. From nick at craig-wood.com Tue Nov 23 16:08:46 2004 From: nick at craig-wood.com (Nick Craig-Wood) Date: Tue Nov 23 16:09:26 2004 Subject: [Python-Dev] Two patches for the new subprocess module Message-ID: <20041123150846.GA22827@craig-wood.com> I have submitted two patches for the subprocess module against cvs HEAD. The first notifies the user of a common mistake and the second adds an IHMO very important utility function. I'd consider the first to be a bug fix and the second a new feature. However a rather nice new feature ;-) ------------------------------------------------------------ [ 1071755 ] raise error for common mistake with subprocess http://python.org/sf/1071755 ------------------------------------------------------------ subprocess has a very easy to mistake error in it - forgetting to pass the command as a list. Eg >>> import subprocess >>> subprocess.call("ls") BaseHTTPServer.py dummy_threading.py pickletools.py socket.py Bastion.py email pipes.py sre.py [...] dummy_thread.py pickle.pyc sndhdr.py zipfile.py 0 >>> subprocess.call("ls", "-l") BaseHTTPServer.py dummy_threading.py pickletools.py socket.py Bastion.py email pipes.py sre.py [...] dummy_thread.py pickle.pyc sndhdr.py zipfile.py 0 >>> # Note no error, warning or anything, but no "ls -l" either And with the patch... >>> subprocess.call("ls", "-l") Traceback (most recent call last): File "<stdin>", line 1, in ? File "subprocess.py", line 428, in call return Popen(*args, **kwargs).wait() File "subprocess.py", line 508, in __init__ raise TypeError("bufsize must be an integer - " TypeError: bufsize must be an integer - did you forget to pass your arguments in a list? >>> ------------------------------------------------------------ [ 1071764 ] a new subprocess.call which raises an error on non-zero rc http://python.org/sf/1071764 ------------------------------------------------------------ The attached patch introduces a 3rd utility function - xcall() to the subprocess module. This function acts just like call but raises errors if the command had a non-zero return code. This saves writing if call(...): raise OSError(...) It is most useful for shell script replacement programming. Checking the return codes of commands called is often forgotten in shell programming. When you've moved up to python because shell is too limiting (usually about 10 lines of shell in my case ;-) you want to make sure that all your commands work so you write robust code. I consider raising an exception to be much more pythonic than checking a return code (ie "Errors should never pass silently" from Zen of Python) Eg # An easy to miss error >>> subprocess.call(["mkdir", "a/b"]) mkdir: cannot create directory `a/b': No such file or directory 1 >>> # user forgot to check non-zero return code # becomes an impossible to miss exception >>> subprocess.xcall(["mkdir", "a/b"]) mkdir: cannot create directory `a/b': No such file or directory Traceback (most recent call last): File "<stdin>", line 1, in ? File "subprocess.py", line 462, in xcall raise CalledProcessError(rc, "Command %s returned non zero exit status" % args[0]) subprocess.CalledProcessError: [Errno 1] Command ['mkdir', 'a/b'] returned non zero exit status >>> See patch for more! Its been tested under python 2.3 on windows and linux. -- Nick Craig-Wood <nick@craig-wood.com> -- http://www.craig-wood.com/nick From hyeshik at gmail.com Tue Nov 23 18:26:12 2004 From: hyeshik at gmail.com (Hye-Shik Chang) Date: Tue Nov 23 18:26:14 2004 Subject: [Python-Dev] py2.4 and gcc4 In-Reply-To: <200411240019.54226.anthony@interlink.com.au> References: <200411240019.54226.anthony@interlink.com.au> Message-ID: <4f0b69dc041123092651196947@mail.gmail.com> On Wed, 24 Nov 2004 00:19:53 +1100, Anthony Baxter <anthony@interlink.com.au> wrote: > Fedora rawhide now ships with a gcc4 preview package - I thought I'd build > current-cvs with it. > > Readline fails to build with > gcc4 -pthread -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -fPIC > -fno-strict-aliasing -I. -I/home/anthony/src/py/pyhead/dist/src/./Include > -I/usr/local/include -I/home/anthony/src/py/pyhead/dist/src/Include > -I/home/anthony/src/py/pyhead/dist/src > -c /home/anthony/src/py/pyhead/dist/src/Modules/readline.c -o > build/temp.linux-i686-2.4/readline.o > /home/anthony/src/py/pyhead/dist/src/Modules/readline.c:97: error: static > declaration of 'history_length' follows non-static declaration > /usr/include/readline/history.h:224: error: previous declaration of > 'history_length' was here history_length is exported by readline and our readline module has another history_length for its interal static variable. I could build readline module by renaming the variable. > test_cfgparser causes a segfault, unfortunately. Since this doesn't > occur with any other compilers that I know of (*) I'm inclined to > blame gcc4, and not bother too much with it further. I'm pretty > short of time, but if someone wants to spend time figuring out > why, let me know and I'll dump a SF bugreport with the stack > trace &c... Seemingly, wrong optimization of gcc4. I was struck by another segfaults from Objects/stringobject.c:1028 and etc. But I could avoid segfaults by disturbing optimizations with some debug outputs. And even full regression test is passed when I built it with -O0. No worry. :) Hye-Shik From rowen at cesmail.net Tue Nov 23 21:47:37 2004 From: rowen at cesmail.net (Russell E. Owen) Date: Tue Nov 23 21:47:43 2004 Subject: [Python-Dev] Re: Two patches for the new subprocess module References: <20041123150846.GA22827@craig-wood.com> Message-ID: <rowen-E0468C.12473623112004@sea.gmane.org> In article <20041123150846.GA22827@craig-wood.com>, Nick Craig-Wood <nick@craig-wood.com> wrote: > I have submitted two patches for the subprocess module against cvs > HEAD. The first notifies the user of a common mistake... > > subprocess has a very easy to mistake error in it - forgetting to pass > the command as a list.... with the patch... > > >>> subprocess.call("ls", "-l") > Traceback (most recent call last): > File "<stdin>", line 1, in ? > File "subprocess.py", line 428, in call > return Popen(*args, **kwargs).wait() > File "subprocess.py", line 508, in __init__ > raise TypeError("bufsize must be an integer - " > TypeError: bufsize must be an integer - did you forget to pass your arguments > in a list? I hope you didn't totally eliminate the ability for Popen and call to accept args as a string? It is a useful feature and one I've been using. If you have broken this, please consider some alternative to your patch. Perhaps changing the default for shell to True if args is a string, False if a list would suffice? I'm not sure this is really the right thing to do on all platforms, but it'd be an easy fix (use shell=None in the argument list and then convert it to True or False as needed). One could imagine other fixes, such as offering one variant each of Popen and call that insists on args=string, leaving the main version to insist on args=list. Regards, -- Russell From bac at OCF.Berkeley.EDU Wed Nov 24 04:02:18 2004 From: bac at OCF.Berkeley.EDU (Brett C.) Date: Wed Nov 24 04:02:27 2004 Subject: [Python-Dev] Current CVS, Cygwin and "make test" In-Reply-To: <20041123124553.GB2096@tishler.net> References: <419E0B4E.7040109@iinet.net.au> <20041119232733.GB1680@tishler.net> <419EABE2.9040804@iinet.net.au> <20041120150637.GA3024@tishler.net> <20041123124553.GB2096@tishler.net> Message-ID: <41A3F9BA.3090702@ocf.berkeley.edu> Jason Tishler wrote: > The test_tcl test case is failing under Cygwin because it expects > loadtk() to fail if the DISPLAY environment variable is not set. > However, under Cygwin we have the following: > > >>> import os, Tkinter > >>> del os.environ['DISPLAY'] > >>> Tkinter.Tcl().loadtk() > >>> > > The above works because Cygwin's Tcl/Tk renders directly on the Windows > GDI instead of going through an X Server. Can I submit a patch to > test_tcl to skip this test as is done for win and darwin? You could, but I just fixed it anyway. =) In revision 1.6 . -Brett From kbk at shore.net Wed Nov 24 07:23:33 2004 From: kbk at shore.net (Kurt B. Kaiser) Date: Wed Nov 24 07:23:38 2004 Subject: [Python-Dev] Weekly Python Patch/Bug Summary Message-ID: <200411240623.iAO6NXE3029528@h006008a7bda6.ne.client2.attbi.com> Patch / Bug Summary ___________________ Patches : 254 open ( +8) / 2700 closed ( +1) / 2954 total ( +9) Bugs : 784 open (+14) / 4629 closed (+15) / 5413 total (+29) RFE : 156 open ( +0) / 135 closed ( +0) / 291 total ( +0) New / Reopened Patches ______________________ os.path.exists returns False if no permission (2004-11-17) http://python.org/sf/1068277 opened by Robert Brewer small update for pdb docs (2004-11-17) CLOSED http://python.org/sf/1068456 opened by Ilya Sandler incomplete support for AF_PACKET in socketmodule.c (2004-11-19) http://python.org/sf/1069624 opened by Gustavo J. A. M. Carneiro xmlrpclib - marshalling new-style classes. (2004-11-20) http://python.org/sf/1070046 opened by Gabriel Pastor Add BLANK_LINE to token.py (2004-11-20) http://python.org/sf/1070218 opened by logistix bug#1021756: more informative error message (2004-11-23) http://python.org/sf/1071739 opened by Christos Georgiou raise error for common mistake with subprocess (2004-11-23) http://python.org/sf/1071755 opened by Nick Craig-Wood a new subprocess.call which raises an error on non-zero rc (2004-11-23) http://python.org/sf/1071764 opened by Nick Craig-Wood Add support for db 4.3 (2004-11-23) http://python.org/sf/1071911 opened by Michal Čihař Patches Closed ______________ small update for pdb docs (2004-11-18) http://python.org/sf/1068456 closed by rhettinger New / Reopened Bugs ___________________ subprocess is not EINTR-safe (2004-11-17) http://python.org/sf/1068268 opened by Peter ?strand linecache.py::update_cache strips directory info from files (2004-11-18) http://python.org/sf/1068477 opened by Tim Leslie confusing new method names for lists (2004-11-18) CLOSED http://python.org/sf/1068590 opened by McErnie TclError not a subclass of StandardError (2004-11-18) http://python.org/sf/1068881 opened by Russell Owen segfault on printing nested sequences of None/Ellipsis (2004-11-19) http://python.org/sf/1069092 opened by Jonas K?lker PyThreadState_SetAsyncExc segfault (2004-11-18) http://python.org/sf/1069160 opened by Tim Peters OS X (Panther) Framework Install causes version mismatch (2004-11-18) http://python.org/sf/1069198 opened by Dusty Harr "Python/C API Reference Manual" lost one Function (2004-11-19) CLOSED http://python.org/sf/1069287 opened by Raise L. Sail C:\Python24\Lib\compileall.py returns False (2004-11-19) http://python.org/sf/1069409 opened by PieterB import on Windows: please call SetErrorMode first (2004-11-19) http://python.org/sf/1069410 opened by Dimitri Papadopoulos Python 2.4c1 does not have universal newline support (2004-11-19) http://python.org/sf/1069433 opened by PieterB setup.py fails - cannot find byteyears.py (2004-11-20) http://python.org/sf/1070121 reopened by bcannon setup.py fails - cannot find byteyears.py (2004-11-20) CLOSED http://python.org/sf/1070121 opened by crescentd endianness detection fails on IRIX 5.3 (2004-11-20) http://python.org/sf/1070140 opened by Georg Schwarz urllib2 authentication redirection error(?) (2004-11-21) http://python.org/sf/1070735 opened by Allan B. Wilson urllib2 authentication redirection error(?) (2004-11-22) http://python.org/sf/1071147 opened by Allan B. Wilson os.walk example for deleting a full tree is sometime wrong (2004-11-22) CLOSED http://python.org/sf/1071087 opened by Olivier Bornet some latex reject the pdfinfo macro while generating html (2004-11-22) http://python.org/sf/1071094 opened by Marc-Antoine Parent Documented grammar for List displays incorrect (testlist) (2004-11-22) http://python.org/sf/1071248 opened by Lenard Lindstrom email: no way to add additional parameter to Content-Type (2004-11-22) CLOSED http://python.org/sf/1071459 opened by Tessa Lau test_sutil fails on cygwin (2004-11-23) http://python.org/sf/1071513 reopened by tebeka test_sutil fails on cygwin (2004-11-23) http://python.org/sf/1071513 opened by Miki Tebeka test_tcl fails on cygwin (2004-11-23) CLOSED http://python.org/sf/1071514 opened by Miki Tebeka test_subprocess fails on cygwin (2004-11-23) http://python.org/sf/1071516 opened by Miki Tebeka moneyfmt recipe in decimal documentation has error (2004-11-23) CLOSED http://python.org/sf/1071566 opened by Anna Ravenscroft coercing decimal to int doesn't work between -1 and 1 (2004-11-23) http://python.org/sf/1071588 opened by Anna Ravenscroft Windows msi doesn't install site-packages directory (2004-11-23) http://python.org/sf/1071594 opened by Anna Ravenscroft configure problem on HP-UX 11.11 (2004-11-23) http://python.org/sf/1071597 opened by Harri Pasanen difflib HTML support description incorrect (2004-11-23) CLOSED http://python.org/sf/1072124 opened by Dan Gass bad arg type to isspace in struct module (2004-11-23) http://python.org/sf/1072182 opened by Greg McFarlane re module segfaulting in large regular expression (2004-11-23) http://python.org/sf/1072259 opened by Erik Demaine Bugs Closed ___________ Obsolete info in Tutorial 9.1 (2004-11-15) http://python.org/sf/1067018 closed by rhettinger Small typo (2004-11-15) http://python.org/sf/1067023 closed by rhettinger subprocess works poorly on Windows with Python 2.3 (2004-10-29) http://python.org/sf/1057048 closed by astrand Overflow error seek()ing with float values > (2 ** 31) - 1 (2004-11-16) http://python.org/sf/1067728 closed by rhettinger Typo about PyErr_WriteUnraisable() (2004-11-14) http://python.org/sf/1066036 closed by rhettinger confusing new method names for lists (2004-11-18) http://python.org/sf/1068590 closed by rhettinger "Python/C API Reference Manual" lost one Function (2004-11-19) http://python.org/sf/1069287 closed by bcannon test_socket fails (2004-11-06) http://python.org/sf/1061429 closed by bcannon setup.py fails - cannot find byteyears.py (2004-11-20) http://python.org/sf/1070121 closed by bcannon setup.py fails - cannot find byteyears.py (2004-11-20) http://python.org/sf/1070121 closed by bcannon os.walk example for deleting a full tree is sometime wrong (2004-11-22) http://python.org/sf/1071087 closed by tim_one Ctrl-C doesn't work with sleepy main thread (2004-01-12) http://python.org/sf/875692 closed by kbk email: no way to add additional parameter to Content-Type (2004-11-23) http://python.org/sf/1071459 closed by bwarsaw test_sutil fails on cygwin (2004-11-23) http://python.org/sf/1071513 closed by jlgijsbers test_tcl fails on cygwin (2004-11-23) http://python.org/sf/1071514 closed by bcannon moneyfmt recipe in decimal documentation has error (2004-11-23) http://python.org/sf/1071566 closed by rhettinger difflib HTML support description incorrect (2004-11-23) http://python.org/sf/1072124 closed by jlgijsbers From nick at craig-wood.com Wed Nov 24 10:55:37 2004 From: nick at craig-wood.com (Nick Craig-Wood) Date: Wed Nov 24 10:55:40 2004 Subject: [Python-Dev] Re: Two patches for the new subprocess module In-Reply-To: <rowen-E0468C.12473623112004@sea.gmane.org> References: <20041123150846.GA22827@craig-wood.com> <rowen-E0468C.12473623112004@sea.gmane.org> Message-ID: <20041124095537.GA29790@craig-wood.com> On Tue, Nov 23, 2004 at 12:47:37PM -0800, Russell E. Owen wrote: > In article <20041123150846.GA22827@craig-wood.com>, > Nick Craig-Wood <nick@craig-wood.com> wrote: > > > I have submitted two patches for the subprocess module against cvs > > HEAD. The first notifies the user of a common mistake... > > > > subprocess has a very easy to mistake error in it - forgetting to pass > > the command as a list.... with the patch... > > > > >>> subprocess.call("ls", "-l") > > Traceback (most recent call last): > > File "<stdin>", line 1, in ? > > File "subprocess.py", line 428, in call > > return Popen(*args, **kwargs).wait() > > File "subprocess.py", line 508, in __init__ > > raise TypeError("bufsize must be an integer - " > > TypeError: bufsize must be an integer - did you forget to pass your arguments > > in a list? > > I hope you didn't totally eliminate the ability for Popen and call to > accept args as a string? It is a useful feature and one I've been using. No, you can still pass *ONE* string as an arg, and that works fine. If you attempt to pass two (not in a list) it raises an exception as above. If you pass two arguments to Popen() or call() the second actually sets the buffer size... > If you have broken this, please consider some alternative to your patch. > > Perhaps changing the default for shell to True if args is a string, > False if a list would suffice? I'm not sure this is really the right > thing to do on all platforms, but it'd be an easy fix (use shell=None in > the argument list and then convert it to True or False as needed). That is a really bad idea IMHO! Its exactly what perl does in its system() built in function, and it is a huge source of security holes in perl scripts. Eg programmer writes system("prog"). Then realises that prog needs an argument so writes system("prog $arg"). This is now a potential security problem if $arg was supplied by a user, because this will be interpreted by the shell. Say the user makes $arg="blah ; rm -rf *" then you are in trouble. If you are thinking about security you'd write system("prog", $arg), but the only way this error is caught in perl is with code review. Back to python - the subprocess module doesn't have this problem. subprocess.call("prog "+arg) won't actually work because it won't find the binary called "prog "+arg. It will just throw an exception at that point - a very definite clue to the programmer that there is a mistake. Eg >>> subprocess.call("ls -l") Traceback (most recent call last): File "<stdin>", line 1, in ? File "subprocess.py", line 447, in call return Popen(*args, **kwargs).wait() File "subprocess.py", line 592, in __init__ errread, errwrite) File "subprocess.py", line 1024, in _execute_child raise child_exception OSError: [Errno 2] No such file or directory >>> Having to write subprocess.call("prog "+arg, shell=True) at least should give the programmer pause for thought, and hopefully the shorter and better subprocess.call(["prog", arg]) would come to mind. -- Nick Craig-Wood <nick@craig-wood.com> -- http://www.craig-wood.com/nick From jlg at dds.nl Wed Nov 24 11:39:11 2004 From: jlg at dds.nl (Johannes Gijsbers) Date: Wed Nov 24 11:36:33 2004 Subject: [Python-Dev] Subversion on SourceForge Message-ID: <20041124103911.GA9168@authsmtp.dds.nl> FIY, I just received this in the monthly update from SourceForge: "For year six, we have a lot of exciting things planned, including UI updates, enhanced tools, new tools, and Subversion support (version control). It will be an exciting year. We can't wait to show you." Cheers, Johannes From jason at tishler.net Wed Nov 24 22:23:42 2004 From: jason at tishler.net (Jason Tishler) Date: Wed Nov 24 22:21:49 2004 Subject: [Python-Dev] Current CVS, Cygwin and "make test" In-Reply-To: <41A3F9BA.3090702@ocf.berkeley.edu> References: <419E0B4E.7040109@iinet.net.au> <20041119232733.GB1680@tishler.net> <419EABE2.9040804@iinet.net.au> <20041120150637.GA3024@tishler.net> <20041123124553.GB2096@tishler.net> <41A3F9BA.3090702@ocf.berkeley.edu> Message-ID: <20041124212342.GA2140@tishler.net> Brett, On Tue, Nov 23, 2004 at 07:02:18PM -0800, Brett C. wrote: > Jason Tishler wrote: > >The above works because Cygwin's Tcl/Tk renders directly on the > >Windows GDI instead of going through an X Server. Can I submit a > >patch to test_tcl to skip this test as is done for win and darwin? > > You could, but I just fixed it anyway. =) In revision 1.6 . Thanks! Jason -- PGP/GPG Key: http://www.tishler.net/jason/pubkey.asc or key servers Fingerprint: 7A73 1405 7F2B E669 C19D 8784 1AFD E4CC ECF4 8EF6 From mal at egenix.com Fri Nov 26 12:23:03 2004 From: mal at egenix.com (M.-A. Lemburg) Date: Fri Nov 26 12:23:06 2004 Subject: [Python-Dev] Multilib strikes back In-Reply-To: <1400815C-3D0C-11D9-96B0-000A95A50FB2@fuhm.net> References: <20041122185211.GL19174@abulafia.devel.redhat.com> <1f7befae0411221104119c1a08@mail.gmail.com> <20041122194939.GN19174@abulafia.devel.redhat.com> <16802.17812.721194.512642@montanaro.dyndns.org> <20041123021914.GC26050@abulafia.devel.redhat.com> <1400815C-3D0C-11D9-96B0-000A95A50FB2@fuhm.net> Message-ID: <41A71217.7090609@egenix.com> James Y Knight wrote: > > On Nov 22, 2004, at 9:19 PM, Mihai Ibanescu wrote: > >> On Mon, Nov 22, 2004 at 02:01:24PM -0600, Skip Montanaro wrote: >> >>> Mihai> How about a _pyconfig_32.h and a _pyconfig_64.h and have >>> Mihai> pyconfig.h include one or the other, depending on the >>> Mihai> environment? >>> >>> How would that be detected? As I understand the original bug report, >>> the >>> user gave gcc a -m32 flag. How would that be reflected in the >>> environment? >> >> >> With the help of Jakub: pyconfig.h would have something close to: >> >> #include <limits.h> >> >> #if CHAR_BIT == 8 && LONG_MAX == 0x7fffffff >> #include "_pyconfig_32.h" >> #elif CHAR_BIT == 8 && LONG_MAX == 0x7fffffffffffffff >> #include "_pyconfig_64.h" >> #else >> #error Unable to detect architecture word length >> #endif >> >> This way, if the compiler is in 32-bit mode, the proper file is included. > > > I think that is the wrong solution. The right solution is to put > pyconfig.h in an arch-specific directory. +1 I wonder why 64-bit Unixes seem to ignore the fact that header files may very well include platform specific settings. Currently, only libs and binary files are separated according to platform. However, this is a OS builder related problem and not one that Python will need to solve. > In current standards, that'd > probably have to be /usr/lib*: e.g. /usr/lib/python2.4/include/ and > /usr/lib64/python2.4/include/. In the proposed multiarch standard > (http://www.linuxbase.org/~taggart/multiarch.html), that would be > /usr/include/i386-linux/ and /usr/include/x86_64-linux/. > > Different, but slightly related: since .py files aren't platform > dependent, /usr/lib* is really the wrong place for them. They should be > going in /usr/share/python2.X/ not /usr/lib*/python2.X/. Obviously the > .so files still need to go in /usr/lib*, though, so that might be a bit > of a trick. That won't work: extensions often place their .so files into their package directories. As a result they need a platforma specific location as well. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Nov 26 2004) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From mal at egenix.com Fri Nov 26 12:37:30 2004 From: mal at egenix.com (M.-A. Lemburg) Date: Fri Nov 26 12:37:33 2004 Subject: [Python-Dev] Python 2.4, MS .NET 1.1 and distutils Message-ID: <41A7157A.7030405@egenix.com> I just tried to compile some extensions on Win XP with .NET 1.1 and Python 2.4c1 installed. It turns out that the distutils recognition logic for finding a suitable C compiler is not able to find the compiler by simply looking at the PATH, INCLUDE and LIB settings of the OS instead of poking around in the registry :-) Now since extensions for 2.4 will have to be built using the same compiler as Python itself (the one that comes with VC7.1 which is the same as the one in the freely downloadable .NET 1.1 SDK), I would suggest to make distutils aware of the compiler by looking on the PATH in case the registry doesn't have the VC7.1 entries. I think this would be a good patch candidate for 2.4.1 - not sure whether it's important enough to squeeze something into 2.4c2. What do you think ? -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Nov 26 2004) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From ncoghlan at iinet.net.au Fri Nov 26 15:21:27 2004 From: ncoghlan at iinet.net.au (Nick Coghlan) Date: Fri Nov 26 15:21:30 2004 Subject: [Python-Dev] String literal concatenation & docstrings Message-ID: <41A73BE7.7020707@iinet.net.au> A c.l.p question about docstring formatting got me curious about something. http://www.python.org/doc/2.3.4/ref/string-catenation.html states that: Multiple adjacent string literals (delimited by whitespace), possibly using different quoting conventions, are allowed, and their meaning is the same as their concatenation. Thus, "hello" 'world' is equivalent to "helloworld". This isn't quite true, since the following doesn't work: def some_func(): """Doc string line 1 (the only line, surprisingly)\n""" """Doc string line 2, except it isn't.""" It seems like an odd quirk that the compile-time concatenation of string literals doesn't work for docstrings. I had a bit of trawl through the docs and the archive with Google, but couldn't find anything that stated whether this behaviour was deliberate or accidental. So, can anyone satisfy my idle curiousity as to whether this was a deliberate design choice, or an accident of the implementation? Cheers, Nick. -- Nick Coghlan | ncoghlan@email.com | Brisbane, Australia --------------------------------------------------------------- http://boredomandlaziness.skystorm.net From theller at python.net Fri Nov 26 15:26:18 2004 From: theller at python.net (Thomas Heller) Date: Fri Nov 26 15:25:29 2004 Subject: [Python-Dev] Python 2.4, MS .NET 1.1 and distutils In-Reply-To: <41A7157A.7030405@egenix.com> (M.'s message of "Fri, 26 Nov 2004 12:37:30 +0100") References: <41A7157A.7030405@egenix.com> Message-ID: <3bywu1o5.fsf@python.net> "M.-A. Lemburg" <mal@egenix.com> writes: > I just tried to compile some extensions on Win XP with .NET 1.1 > and Python 2.4c1 installed. > > It turns out that the distutils recognition logic for finding > a suitable C compiler is not able to find the compiler by simply > looking at the PATH, INCLUDE and LIB settings of the OS instead > of poking around in the registry :-) > > Now since extensions for 2.4 will have to be built using the > same compiler as Python itself (the one that comes with VC7.1 > which is the same as the one in the freely downloadable > .NET 1.1 SDK), I would suggest to make distutils aware of the > compiler by looking on the PATH in case the registry doesn't > have the VC7.1 entries. I always wondered if non-present registry entries wouldn't mean that the MSVC installation is broken. OTOH, I always used the official MS IDEs, and never tried their free compilers - is this desire related to the free ones? Thomas From jim at zope.com Fri Nov 26 16:01:06 2004 From: jim at zope.com (Jim Fulton) Date: Fri Nov 26 16:01:10 2004 Subject: [Python-Dev] Python 2.4, MS .NET 1.1 and distutils In-Reply-To: <41A7157A.7030405@egenix.com> References: <41A7157A.7030405@egenix.com> Message-ID: <41A74532.1050201@zope.com> M.-A. Lemburg wrote: > I just tried to compile some extensions on Win XP with .NET 1.1 > and Python 2.4c1 installed. > > It turns out that the distutils recognition logic for finding > a suitable C compiler is not able to find the compiler by simply > looking at the PATH, INCLUDE and LIB settings of the OS instead > of poking around in the registry :-) > > Now since extensions for 2.4 will have to be built using the > same compiler as Python itself (the one that comes with VC7.1 > which is the same as the one in the freely downloadable > .NET 1.1 SDK), I would suggest to make distutils aware of the > compiler by looking on the PATH in case the registry doesn't > have the VC7.1 entries. > > I think this would be a good patch candidate for 2.4.1 +1 - not > sure whether it's important enough to squeeze something into > 2.4c2. I think it isn't. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From mwh at python.net Fri Nov 26 16:06:01 2004 From: mwh at python.net (Michael Hudson) Date: Fri Nov 26 16:06:02 2004 Subject: [Python-Dev] String literal concatenation & docstrings In-Reply-To: <41A73BE7.7020707@iinet.net.au> (Nick Coghlan's message of "Sat, 27 Nov 2004 00:21:27 +1000") References: <41A73BE7.7020707@iinet.net.au> Message-ID: <2mpt20veee.fsf@starship.python.net> Nick Coghlan <ncoghlan@iinet.net.au> writes: > A c.l.p question about docstring formatting got me curious about something. > > http://www.python.org/doc/2.3.4/ref/string-catenation.html states that: > Multiple adjacent string literals (delimited by whitespace), possibly using > different quoting conventions, are allowed, and their meaning is the same as > their concatenation. Thus, "hello" 'world' is equivalent to "helloworld". > > This isn't quite true, since the following doesn't work: > > def some_func(): > """Doc string line 1 (the only line, surprisingly)\n""" > """Doc string line 2, except it isn't.""" I haven't actually checked or anything rash like that, but I'd imagine the answer is something like: The two strings are separate statements as far as the parser is concerned, and the "concatenating adjacent strings" thing only happens within an expression. You can do this: >>> "con"\ ... "cat" 'concat' > So, can anyone satisfy my idle curiousity as to whether this was a > deliberate design choice, or an accident of the implementation? Well, it surprises me not at all. Cheers, mwh -- ARTHUR: Why should he want to know where his towel is? FORD: Everybody should know where his towel is. ARTHUR: I think your head's come undone. -- The Hitch-Hikers Guide to the Galaxy, Episode 7 From ncoghlan at iinet.net.au Fri Nov 26 16:19:10 2004 From: ncoghlan at iinet.net.au (Nick Coghlan) Date: Fri Nov 26 16:19:18 2004 Subject: [Python-Dev] String literal concatenation & docstrings In-Reply-To: <2mpt20veee.fsf@starship.python.net> References: <41A73BE7.7020707@iinet.net.au> <2mpt20veee.fsf@starship.python.net> Message-ID: <41A7496E.9020309@iinet.net.au> Michael Hudson wrote: > Nick Coghlan <ncoghlan@iinet.net.au> writes: > I haven't actually checked or anything rash like that, but I'd imagine > the answer is something like: > > The two strings are separate statements as far as the parser is > concerned, and the "concatenating adjacent strings" thing only > happens within an expression. That would certainly be a sensible explanation. The only time I've ever actually made use of the feature is when assigning a long string, and even then only rarely (I'm more likely to use triple quotes and left align the whole thing) > You can do this: >>>>"con"\ > > ... "cat" > 'concat' Which actually does work for combining multiple strings into a single docstring. >>So, can anyone satisfy my idle curiousity as to whether this was a >>deliberate design choice, or an accident of the implementation? > > Well, it surprises me not at all. I think the key distinction I'd missed was that in the doc string case, the two strings were actually separate statements. Once that distinction is noted, the behaviour is, as you say, unsurprising. It also makes it obvious why escaping the newline has the effect it does. Cheers, Nick. -- Nick Coghlan | ncoghlan@email.com | Brisbane, Australia --------------------------------------------------------------- http://boredomandlaziness.skystorm.net From martin at v.loewis.de Fri Nov 26 17:27:42 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri Nov 26 17:27:42 2004 Subject: [Python-Dev] Python 2.4, MS .NET 1.1 and distutils In-Reply-To: <41A7157A.7030405@egenix.com> References: <41A7157A.7030405@egenix.com> Message-ID: <41A7597E.2090805@v.loewis.de> M.-A. Lemburg wrote: > Now since extensions for 2.4 will have to be built using the > same compiler as Python itself (the one that comes with VC7.1 > which is the same as the one in the freely downloadable > .NET 1.1 SDK) My understanding is that these are not exactly the same compilers; I recall that the freely-redistributable one lacks support for optimization. Regards, Martin From ronaldoussoren at mac.com Fri Nov 26 17:36:54 2004 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Fri Nov 26 17:37:03 2004 Subject: [Python-Dev] Python 2.4, MS .NET 1.1 and distutils In-Reply-To: <41A7597E.2090805@v.loewis.de> References: <41A7157A.7030405@egenix.com> <41A7597E.2090805@v.loewis.de> Message-ID: <61A404CD-3FC9-11D9-B357-000D93AD379E@mac.com> On 26-nov-04, at 17:27, Martin v. L?wis wrote: > M.-A. Lemburg wrote: >> Now since extensions for 2.4 will have to be built using the >> same compiler as Python itself (the one that comes with VC7.1 >> which is the same as the one in the freely downloadable >> .NET 1.1 SDK) > > My understanding is that these are not exactly the same compilers; > I recall that the freely-redistributable one lacks support for > optimization. That's not what their download page says: <quote href="http://msdn.microsoft.com/visualc/vctoolkit2003/"> Microsoft Visual C++ Toolkit 2003 The Microsoft Visual C++ Toolkit 2003 includes the core tools developers need to compile and link C++-based applications for Windows and the .NET Common Language Runtime: ? Microsoft C/C++ Optimizing Compiler and Linker.? These are the same compiler and linker that ship with Visual Studio .NET 2003 Professional! </quote> Ronald -- X|support bv http://www.xsupport.nl/ T: +31 610271479 F: +31 204416173 From mal at egenix.com Fri Nov 26 17:49:16 2004 From: mal at egenix.com (M.-A. Lemburg) Date: Fri Nov 26 17:49:15 2004 Subject: [Python-Dev] Python 2.4, MS .NET 1.1 and distutils In-Reply-To: <3bywu1o5.fsf@python.net> References: <41A7157A.7030405@egenix.com> <3bywu1o5.fsf@python.net> Message-ID: <41A75E8C.3030301@egenix.com> Thomas Heller wrote: > "M.-A. Lemburg" <mal@egenix.com> writes: > > >>I just tried to compile some extensions on Win XP with .NET 1.1 >>and Python 2.4c1 installed. >> >>It turns out that the distutils recognition logic for finding >>a suitable C compiler is not able to find the compiler by simply >>looking at the PATH, INCLUDE and LIB settings of the OS instead >>of poking around in the registry :-) >> >>Now since extensions for 2.4 will have to be built using the >>same compiler as Python itself (the one that comes with VC7.1 >>which is the same as the one in the freely downloadable >>.NET 1.1 SDK), I would suggest to make distutils aware of the >>compiler by looking on the PATH in case the registry doesn't >>have the VC7.1 entries. > > I always wondered if non-present registry entries wouldn't mean that the > MSVC installation is broken. Not at all: their C compiler works just fine once you setup the OS environment by loading the sdkvars.bat file. > OTOH, I always used the official MS IDEs, > and never tried their free compilers - is this desire related to the > free ones? Yes. distutils doesn't need any IDE to compile extensions, just a compiler and a command line :-) -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Nov 26 2004) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From theller at python.net Fri Nov 26 17:53:05 2004 From: theller at python.net (Thomas Heller) Date: Fri Nov 26 17:52:15 2004 Subject: [Python-Dev] Python 2.4, MS .NET 1.1 and distutils In-Reply-To: <61A404CD-3FC9-11D9-B357-000D93AD379E@mac.com> (Ronald Oussoren's message of "Fri, 26 Nov 2004 17:36:54 +0100") References: <41A7157A.7030405@egenix.com> <41A7597E.2090805@v.loewis.de> <61A404CD-3FC9-11D9-B357-000D93AD379E@mac.com> Message-ID: <653ssgb2.fsf@python.net> Ronald Oussoren <ronaldoussoren@mac.com> writes: > On 26-nov-04, at 17:27, Martin v. L?wis wrote: > >> M.-A. Lemburg wrote: >>> Now since extensions for 2.4 will have to be built using the >>> same compiler as Python itself (the one that comes with VC7.1 >>> which is the same as the one in the freely downloadable >>> .NET 1.1 SDK) >> >> My understanding is that these are not exactly the same compilers; >> I recall that the freely-redistributable one lacks support for >> optimization. > > That's not what their download page says: > > <quote href="http://msdn.microsoft.com/visualc/vctoolkit2003/"> > Microsoft Visual C++ Toolkit 2003 > > The Microsoft Visual C++ Toolkit 2003 includes the core tools > developers need to compile and link C++-based applications for > Windows and the .NET Common Language Runtime: > > ? Microsoft C/C++ Optimizing Compiler and Linker.? These > are the same compiler and linker that ship with Visual Studio > .NET 2003 Professional! > </quote> Wasn't the difference that no msvcrt import libraries are included, neither the license to redistribute the msvcrt runtime dlls (although the missing license probably doesn't matter, because they are already in the Python distribution)? Thomas From anthony at interlink.com.au Fri Nov 26 17:52:23 2004 From: anthony at interlink.com.au (Anthony Baxter) Date: Fri Nov 26 17:52:48 2004 Subject: [Python-Dev] Python 2.4, MS .NET 1.1 and distutils In-Reply-To: <41A7157A.7030405@egenix.com> References: <41A7157A.7030405@egenix.com> Message-ID: <200411270352.24397.anthony@interlink.com.au> On Friday 26 November 2004 22:37, M.-A. Lemburg wrote: > I think this would be a good patch candidate for 2.4.1 - not > sure whether it's important enough to squeeze something into > 2.4c2. I don't think it's important enough for 2.4c2/2.4 final, but I'd be happy to see this in 2.4.1. An update: as far as I know, we're still without a fix for the problem Tim identified last week - he seemed to think that it wasn't dangerous enough that we'd need a 2.4c2. I don't know of any other reasons that we'd want a c2, so I'll probably go with the final release next, unless someone has a strong reason not to do so. I'm not sure whether the bug Tim identified is bad enough to hold up 2.4 final for - as far as I recall (I've just switched mailers, and haven't imported my old mail folders yet) it was a function that no-one was using... Anthony From mal at egenix.com Fri Nov 26 18:07:45 2004 From: mal at egenix.com (M.-A. Lemburg) Date: Fri Nov 26 18:07:45 2004 Subject: [Python-Dev] Python 2.4, MS .NET 1.1 and distutils In-Reply-To: <653ssgb2.fsf@python.net> References: <41A7157A.7030405@egenix.com> <41A7597E.2090805@v.loewis.de> <61A404CD-3FC9-11D9-B357-000D93AD379E@mac.com> <653ssgb2.fsf@python.net> Message-ID: <41A762E1.7040404@egenix.com> Thomas Heller wrote: > Ronald Oussoren <ronaldoussoren@mac.com> writes: > > >>On 26-nov-04, at 17:27, Martin v. L?wis wrote: >> >> >>>M.-A. Lemburg wrote: >>> >>>>Now since extensions for 2.4 will have to be built using the >>>>same compiler as Python itself (the one that comes with VC7.1 >>>>which is the same as the one in the freely downloadable >>>>.NET 1.1 SDK) >>> >>>My understanding is that these are not exactly the same compilers; >>>I recall that the freely-redistributable one lacks support for >>>optimization. >> >>That's not what their download page says: >> >><quote href="http://msdn.microsoft.com/visualc/vctoolkit2003/"> >>Microsoft Visual C++ Toolkit 2003 >> >> The Microsoft Visual C++ Toolkit 2003 includes the core tools >> developers need to compile and link C++-based applications for >> Windows and the .NET Common Language Runtime: >> >> ? Microsoft C/C++ Optimizing Compiler and Linker. These >> are the same compiler and linker that ship with Visual Studio >> .NET 2003 Professional! >></quote> The compiler works happily with the optimization flags... > Wasn't the difference that no msvcrt import libraries are included, > neither the license to redistribute the msvcrt runtime dlls (although > the missing license probably doesn't matter, because they are already in > the Python distribution)? ... and links again MSVCRT.LIB which is included in .NET 1.1. The DLLs come with the .NET framework, so there's no need to redistribute them. Anyway, the point is not creating binaries that you distribute, but simply to be able to compile extensions yourself as necessary. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Nov 26 2004) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From ncoghlan at iinet.net.au Fri Nov 26 18:19:08 2004 From: ncoghlan at iinet.net.au (Nick Coghlan) Date: Fri Nov 26 18:19:15 2004 Subject: [Python-Dev] Python 2.4, MS .NET 1.1 and distutils In-Reply-To: <653ssgb2.fsf@python.net> References: <41A7157A.7030405@egenix.com> <41A7597E.2090805@v.loewis.de> <61A404CD-3FC9-11D9-B357-000D93AD379E@mac.com> <653ssgb2.fsf@python.net> Message-ID: <41A7658C.80504@iinet.net.au> Thomas Heller wrote: > Wasn't the difference that no msvcrt import libraries are included, > neither the license to redistribute the msvcrt runtime dlls (although > the missing license probably doesn't matter, because they are already in > the Python distribution)? The main elements needed are the free .NET SDK (which provides a complete C++ toolchain, but has a non-optimising C compiler and a statically-linked only C runtime) and the Visual C++ Toolkit (which replaces the .NET SDK compiler and linker with ones which are actually useful). The toolkit is useless on its own, since it includes the compiler & linker only, and no other required build tools (like, oh, say, nmake). I believe it's also necessary to install the platform SDK. I know I needed it to get the "windows.h" header (since I was building Python itself, rather than an exension) - I think it also included the stub libraries to link to the MSVCRT DLL's. Unfortunately, the Windows partition I had all this set up on died a couple of months back, or I'd be able to just write up a summary of what was required :( Cheers, Nick. -- Nick Coghlan | ncoghlan@email.com | Brisbane, Australia --------------------------------------------------------------- http://boredomandlaziness.skystorm.net From martin at v.loewis.de Fri Nov 26 18:57:11 2004 From: martin at v.loewis.de (=?windows-1252?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri Nov 26 18:57:10 2004 Subject: [Python-Dev] Python 2.4, MS .NET 1.1 and distutils In-Reply-To: <41A762E1.7040404@egenix.com> References: <41A7157A.7030405@egenix.com> <41A7597E.2090805@v.loewis.de> <61A404CD-3FC9-11D9-B357-000D93AD379E@mac.com> <653ssgb2.fsf@python.net> <41A762E1.7040404@egenix.com> Message-ID: <41A76E77.6000901@v.loewis.de> M.-A. Lemburg wrote: >> Wasn't the difference that no msvcrt import libraries are included, >> neither the license to redistribute the msvcrt runtime dlls (although >> the missing license probably doesn't matter, because they are already in >> the Python distribution)? > > > ... and links again MSVCRT.LIB which is included in .NET 1.1. Python 2.4 is linked against msvcr71.dll, though. > The DLLs come with the .NET framework, so there's no need to > redistribute them. Not msvcr71.dll. > Anyway, the point is not creating binaries that you distribute, > but simply to be able to compile extensions yourself as necessary. Yes, but can you also *link* correctly? Regards, Martin From jim at zope.com Fri Nov 26 19:12:03 2004 From: jim at zope.com (Jim Fulton) Date: Fri Nov 26 19:12:08 2004 Subject: [Python-Dev] Python 2.4, MS .NET 1.1 and distutils In-Reply-To: <41A762E1.7040404@egenix.com> References: <41A7157A.7030405@egenix.com> <41A7597E.2090805@v.loewis.de> <61A404CD-3FC9-11D9-B357-000D93AD379E@mac.com> <653ssgb2.fsf@python.net> <41A762E1.7040404@egenix.com> Message-ID: <41A771F3.3060006@zope.com> M.-A. Lemburg wrote: ... > Anyway, the point is not creating binaries that you distribute, > but simply to be able to compile extensions yourself as necessary. Or to distribute sources, especially via cvs or svn that *other* people can compile. Even though I don't use windows myself, it would make my life a lot easier if people could compile Zope exptensions themselves rather than waiting for us or someone else to build windows binaries for them. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From mal at egenix.com Fri Nov 26 19:41:37 2004 From: mal at egenix.com (M.-A. Lemburg) Date: Fri Nov 26 19:41:39 2004 Subject: [Python-Dev] Python 2.4, MS .NET 1.1 and distutils In-Reply-To: <41A76E77.6000901@v.loewis.de> References: <41A7157A.7030405@egenix.com> <41A7597E.2090805@v.loewis.de> <61A404CD-3FC9-11D9-B357-000D93AD379E@mac.com> <653ssgb2.fsf@python.net> <41A762E1.7040404@egenix.com> <41A76E77.6000901@v.loewis.de> Message-ID: <41A778E1.7060006@egenix.com> Martin v. L?wis wrote: > M.-A. Lemburg wrote: > >>> Wasn't the difference that no msvcrt import libraries are included, >>> neither the license to redistribute the msvcrt runtime dlls (although >>> the missing license probably doesn't matter, because they are already in >>> the Python distribution)? >> >> >> >> ... and links again MSVCRT.LIB which is included in .NET 1.1. > > > Python 2.4 is linked against msvcr71.dll, though. The MSVCRT.LIB from .NET 1.1 is an import lib for msvcr71.dll. >> The DLLs come with the .NET framework, so there's no need to >> redistribute them. > > Not msvcr71.dll. It was installed as part of the SDK that came MS update (v1.1.4322). >> Anyway, the point is not creating binaries that you distribute, >> but simply to be able to compile extensions yourself as necessary. > > Yes, but can you also *link* correctly? Yes. HelloWorld.c compiles&links just fine - even the optimized version :-) -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Nov 26 2004) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From bac at OCF.Berkeley.EDU Fri Nov 26 20:56:05 2004 From: bac at OCF.Berkeley.EDU (Brett C.) Date: Fri Nov 26 20:56:13 2004 Subject: [Python-Dev] String literal concatenation & docstrings In-Reply-To: <41A7496E.9020309@iinet.net.au> References: <41A73BE7.7020707@iinet.net.au> <2mpt20veee.fsf@starship.python.net> <41A7496E.9020309@iinet.net.au> Message-ID: <41A78A55.8030802@ocf.berkeley.edu> Nick Coghlan wrote: > Michael Hudson wrote: > >> Nick Coghlan <ncoghlan@iinet.net.au> writes: >> I haven't actually checked or anything rash like that, but I'd imagine >> the answer is something like: >> >> The two strings are separate statements as far as the parser is >> concerned, and the "concatenating adjacent strings" thing only >> happens within an expression. > > > That would certainly be a sensible explanation. The only time I've ever > actually made use of the feature is when assigning a long string, and > even then only rarely (I'm more likely to use triple quotes and left > align the whole thing) > And the sensible explanation is correct. Just checked out the compiler and the string concatenation (in parsestrplus()) takes a node and then proceeds to concatenate all of its children that are strings right in a row starting at child 0. With statements this won't trigger anything since the statements will only have the string as their child, unlike an expression, which will just have all the string pieces. [SNIP] > I think the key distinction I'd missed was that in the doc string case, > the two strings were actually separate statements. Once that distinction > is noted, the behaviour is, as you say, unsurprising. It also makes it > obvious why escaping the newline has the effect it does. > Should probably change the wording on that unless people actually want the literal string concatenation to work with statements (docstrings seem like the only place that would be reasonable) unless you want to start allowing print statements to have a string part span multiple lines. =) -Brett From mal at egenix.com Sat Nov 27 00:26:38 2004 From: mal at egenix.com (M.-A. Lemburg) Date: Sat Nov 27 00:26:36 2004 Subject: [Python-Dev] Python 2.4, MS .NET 1.1 and distutils In-Reply-To: <41A762E1.7040404@egenix.com> References: <41A7157A.7030405@egenix.com> <41A7597E.2090805@v.loewis.de> <61A404CD-3FC9-11D9-B357-000D93AD379E@mac.com> <653ssgb2.fsf@python.net> <41A762E1.7040404@egenix.com> Message-ID: <41A7BBAE.6080907@egenix.com> M.-A. Lemburg wrote: > Thomas Heller wrote: > >> Ronald Oussoren <ronaldoussoren@mac.com> writes: >> >> >>> On 26-nov-04, at 17:27, Martin v. L?wis wrote: >>> >>> >>>> M.-A. Lemburg wrote: >>>> >>>>> Now since extensions for 2.4 will have to be built using the >>>>> same compiler as Python itself (the one that comes with VC7.1 >>>>> which is the same as the one in the freely downloadable >>>>> .NET 1.1 SDK) >>>> >>>> >>>> My understanding is that these are not exactly the same compilers; >>>> I recall that the freely-redistributable one lacks support for >>>> optimization. >>> >>> >>> That's not what their download page says: >>> >>> <quote href="http://msdn.microsoft.com/visualc/vctoolkit2003/"> >>> Microsoft Visual C++ Toolkit 2003 >>> >>> The Microsoft Visual C++ Toolkit 2003 includes the core tools >>> developers need to compile and link C++-based applications for >>> Windows and the .NET Common Language Runtime: >>> >>> ? Microsoft C/C++ Optimizing Compiler and Linker. These >>> are the same compiler and linker that ship with Visual Studio >>> .NET 2003 Professional! >>> </quote> > > > The compiler works happily with the optimization flags... After doing some extra checks, looking at the generated assembler and trying the /O2 flag, I have to correct this: even though the compiler from .NET 1.1 supports the /G optimization flags it doesn't seem to do any of the more complicated optimizations which you get with /O2. Mike Fletcher told me that the Microsoft Visual C++ Toolkit 2003 comes with the optimizing version of the compiler which is also the one mentioned in the quote. He also pointed me at: http://www.thefreecountry.com/compilers/cpp.shtml So in summary you need these things if you want an optimizing compiler for Python 2.4 extensions: * MS VC++ Toolkit 2003 * MS Platform SDK (if you want to build Windows specific extensions) In any, the compiler uses the same command line name, so a PATH based search for the compiler would provide a useful extension for distutils. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Nov 27 2004) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From martin at v.loewis.de Sat Nov 27 02:00:35 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat Nov 27 02:00:35 2004 Subject: [Python-Dev] Python 2.4, MS .NET 1.1 and distutils In-Reply-To: <41A771F3.3060006@zope.com> References: <41A7157A.7030405@egenix.com> <41A7597E.2090805@v.loewis.de> <61A404CD-3FC9-11D9-B357-000D93AD379E@mac.com> <653ssgb2.fsf@python.net> <41A762E1.7040404@egenix.com> <41A771F3.3060006@zope.com> Message-ID: <41A7D1B3.7070306@v.loewis.de> Jim Fulton wrote: > Or to distribute sources, especially via cvs or svn that *other* > people can compile. Even though I don't use windows myself, it > would make my life a lot easier if people could compile Zope > exptensions themselves rather than waiting for us or someone else > to build windows binaries for them. Of course, in that case, it will be sufficient if the other people have VC.NET 2003 installed. Distutils will properly locate that installation, and it will then produce proper extension modules. Regards, Martin From carribeiro at gmail.com Sat Nov 27 04:53:48 2004 From: carribeiro at gmail.com (Carlos Ribeiro) Date: Sat Nov 27 04:53:53 2004 Subject: [Python-Dev] String literal concatenation & docstrings In-Reply-To: <41A78A55.8030802@ocf.berkeley.edu> References: <41A73BE7.7020707@iinet.net.au> <2mpt20veee.fsf@starship.python.net> <41A7496E.9020309@iinet.net.au> <41A78A55.8030802@ocf.berkeley.edu> Message-ID: <864d370904112619532ee559d4@mail.gmail.com> On Fri, 26 Nov 2004 11:56:05 -0800, Brett C. <bac@ocf.berkeley.edu> wrote: > Should probably change the wording on that unless people actually want the > literal string concatenation to work with statements (docstrings seem like the > only place that would be reasonable) unless you want to start allowing print > statements to have a string part span multiple lines. =) It means that: print "this line continues" "on the next line" does not work, while the following works: a = "this line continues" "on the next line" Kind of weird, but anyway, that's not a common idiom. One more reason to use triple-quoted-strings when printing long strings. -- Carlos Ribeiro Consultoria em Projetos blog: http://rascunhosrotos.blogspot.com blog: http://pythonnotes.blogspot.com mail: carribeiro@gmail.com mail: carribeiro@yahoo.com From orbitz at drorbitz.ath.cx Sat Nov 27 05:01:24 2004 From: orbitz at drorbitz.ath.cx (ort) Date: Sat Nov 27 05:01:32 2004 Subject: [Python-Dev] String literal concatenation & docstrings In-Reply-To: <864d370904112619532ee559d4@mail.gmail.com> References: <41A73BE7.7020707@iinet.net.au> <2mpt20veee.fsf@starship.python.net> <41A7496E.9020309@iinet.net.au> <41A78A55.8030802@ocf.berkeley.edu> <864d370904112619532ee559d4@mail.gmail.com> Message-ID: <41A7FC14.8090209@drorbitz.ath.cx> Like anything, if you need to wrap a statement around multiple lines, you surround it in ()'s Now the question is why does: >>> def foo(): ... ("""blah""" ... """fejlfe""") ... pass ... >>> help(foo) Not show that as the doc string. Just because it has () doesn't mean it evaluates to anything other than a string as far as I know. Carlos Ribeiro wrote: >On Fri, 26 Nov 2004 11:56:05 -0800, Brett C. <bac@ocf.berkeley.edu> wrote: > > >>Should probably change the wording on that unless people actually want the >>literal string concatenation to work with statements (docstrings seem like the >>only place that would be reasonable) unless you want to start allowing print >>statements to have a string part span multiple lines. =) >> >> > >It means that: > > print "this line continues" > "on the next line" > >does not work, while the following works: > > a = "this line continues" > "on the next line" > >Kind of weird, but anyway, that's not a common idiom. One more reason >to use triple-quoted-strings when printing long strings. > > > From goodger at python.org Sat Nov 27 16:07:15 2004 From: goodger at python.org (David Goodger) Date: Sat Nov 27 16:07:25 2004 Subject: [Python-Dev] Re: String literal concatenation & docstrings In-Reply-To: <864d370904112619532ee559d4@mail.gmail.com> References: <41A73BE7.7020707@iinet.net.au> <2mpt20veee.fsf@starship.python.net> <41A7496E.9020309@iinet.net.au> <41A78A55.8030802@ocf.berkeley.edu> <864d370904112619532ee559d4@mail.gmail.com> Message-ID: <41A89823.1040403@python.org> [Carlos Ribeiro] > while the following works: > > a = "this line continues" > "on the next line" Are you sure about that? Doesn't work on my machine: $ cat x.py a = "this line continues " "on the next line" $ python x.py File "x.py", line 2 "on the next line" ^ SyntaxError: invalid syntax If you add a trailing backslash, it does work: $ cat x2.py a = "this line continues " \ "on the next line" print a $ python x2.py this line continues on the next line > Kind of weird Not weird at all ;-) -- David Goodger <http://python.net/~goodger> -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 256 bytes Desc: OpenPGP digital signature Url : http://mail.python.org/pipermail/python-dev/attachments/20041127/b6e5ec3c/signature.pgp From jim at zope.com Sat Nov 27 16:38:45 2004 From: jim at zope.com (Jim Fulton) Date: Sat Nov 27 16:38:52 2004 Subject: [Python-Dev] Python 2.4, MS .NET 1.1 and distutils In-Reply-To: <41A7D1B3.7070306@v.loewis.de> References: <41A7157A.7030405@egenix.com> <41A7597E.2090805@v.loewis.de> <61A404CD-3FC9-11D9-B357-000D93AD379E@mac.com> <653ssgb2.fsf@python.net> <41A762E1.7040404@egenix.com> <41A771F3.3060006@zope.com> <41A7D1B3.7070306@v.loewis.de> Message-ID: <41A89F85.9080703@zope.com> Martin v. L?wis wrote: > Jim Fulton wrote: > >> Or to distribute sources, especially via cvs or svn that *other* >> people can compile. Even though I don't use windows myself, it >> would make my life a lot easier if people could compile Zope >> exptensions themselves rather than waiting for us or someone else >> to build windows binaries for them. > > > Of course, in that case, it will be sufficient if the other people > have VC.NET 2003 installed. Is that free? If not, that's a big "if". Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From astrand at lysator.liu.se Sat Nov 27 17:09:05 2004 From: astrand at lysator.liu.se (Peter Astrand) Date: Sat Nov 27 17:09:13 2004 Subject: [Python-Dev] Python 2.4, MS .NET 1.1 and distutils In-Reply-To: <41A7597E.2090805@v.loewis.de> References: <41A7157A.7030405@egenix.com> <41A7597E.2090805@v.loewis.de> Message-ID: <Pine.GSO.4.51L2.0411270847260.28277@koeberg.lysator.liu.se> On Fri, 26 Nov 2004, "Martin v. L?wis" wrote: > M.-A. Lemburg wrote: > > Now since extensions for 2.4 will have to be built using the > > same compiler as Python itself (the one that comes with VC7.1 > > which is the same as the one in the freely downloadable > > .NET 1.1 SDK) > > My understanding is that these are not exactly the same compilers; > I recall that the freely-redistributable one lacks support for > optimization. On the contrary: The commercial, boxed "Visual Studio .NET 2003 Standard" lacks support for optimization, but the free "Visual C++ Toolkit 2003" can optimize. It is possible to use the free, optimizing compiler with Visual Studio .NET 2003 Standard, though: See http://www.sawtoothdistortion.com/Articles/FreeOptimizingCompiler.html /Peter ?strand <astrand@lysator.liu.se> From gvanrossum at gmail.com Sat Nov 27 18:13:25 2004 From: gvanrossum at gmail.com (Guido van Rossum) Date: Sat Nov 27 18:13:29 2004 Subject: [Python-Dev] String literal concatenation & docstrings In-Reply-To: <864d370904112619532ee559d4@mail.gmail.com> References: <41A73BE7.7020707@iinet.net.au> <2mpt20veee.fsf@starship.python.net> <41A7496E.9020309@iinet.net.au> <41A78A55.8030802@ocf.berkeley.edu> <864d370904112619532ee559d4@mail.gmail.com> Message-ID: <ca471dc20411270913780b7e63@mail.gmail.com> > It means that: > > print "this line continues" > "on the next line" > > does not work, while the following works: > > a = "this line continues" > "on the next line" As has been pointed out already, it doesn't. The right way to look at this is not using the statement/expression distinction, but to look at whether the newline between the two literals is significant to the parser or not. A significant newline ends a statement; an insignificant one is equivalent to a space. The rule is that newlines (outside string quotes anyway) are significant unless either escaped with \, or contained within matching parentheses. So if you want to print or assign a long string broken across multiple lines without having to use triple-quoted strings (which have their own set of issues), you can write this: print ("this line continues" " on the next line") Note that I fixed a common buglet in your example: the words "continues" and "on" should be separated by a space. Not important for the example, of course, but (having used this idiom and similar ones a lot) something to keep in mind when splitting a long string across a line this way. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From bac at OCF.Berkeley.EDU Sat Nov 27 20:08:34 2004 From: bac at OCF.Berkeley.EDU (Brett C.) Date: Sat Nov 27 20:08:42 2004 Subject: [Python-Dev] String literal concatenation & docstrings In-Reply-To: <ca471dc20411270913780b7e63@mail.gmail.com> References: <41A73BE7.7020707@iinet.net.au> <2mpt20veee.fsf@starship.python.net> <41A7496E.9020309@iinet.net.au> <41A78A55.8030802@ocf.berkeley.edu> <864d370904112619532ee559d4@mail.gmail.com> <ca471dc20411270913780b7e63@mail.gmail.com> Message-ID: <41A8D0B2.6060800@ocf.berkeley.edu> Guido van Rossum wrote: >>It means that: >> >> print "this line continues" >> "on the next line" >> >>does not work, while the following works: >> >> a = "this line continues" >> "on the next line" > > > As has been pointed out already, it doesn't. > > The right way to look at this is not using the statement/expression > distinction, but to look at whether the newline between the two > literals is significant to the parser or not. A significant newline > ends a statement; an insignificant one is equivalent to a space. The > rule is that newlines (outside string quotes anyway) are significant > unless either escaped with \, or contained within matching > parentheses. > So how is this for new wording? "Multiple adjacent string literals (delimited by whitespace), possibly using different quoting conventions, are allowed, and their meaning is the same as their concatenation as long as the newline separating them is not signifcant to the parser." From gvanrossum at gmail.com Sat Nov 27 22:05:42 2004 From: gvanrossum at gmail.com (Guido van Rossum) Date: Sat Nov 27 22:05:46 2004 Subject: [Python-Dev] String literal concatenation & docstrings In-Reply-To: <41A8D0B2.6060800@ocf.berkeley.edu> References: <41A73BE7.7020707@iinet.net.au> <2mpt20veee.fsf@starship.python.net> <41A7496E.9020309@iinet.net.au> <41A78A55.8030802@ocf.berkeley.edu> <864d370904112619532ee559d4@mail.gmail.com> <ca471dc20411270913780b7e63@mail.gmail.com> <41A8D0B2.6060800@ocf.berkeley.edu> Message-ID: <ca471dc2041127130529ecdf98@mail.gmail.com> > So how is this for new wording? > > "Multiple adjacent string literals (delimited by whitespace), possibly using > different quoting conventions, are allowed, and their meaning is the same as > their concatenation as long as the newline separating them is not signifcant to > the parser." I'm not sure it needs clarifying; it's the reference manual after all, not a tutorial. I'd rather let the grammar speak for itself; there's no ambiguity in that, and the words are just there to clarify the *semantics*. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From python at rcn.com Sat Nov 27 22:10:14 2004 From: python at rcn.com (Raymond Hettinger) Date: Sat Nov 27 22:13:28 2004 Subject: [Python-Dev] String literal concatenation & docstrings In-Reply-To: <ca471dc2041127130529ecdf98@mail.gmail.com> Message-ID: <001001c4d4c5$7d2d8ec0$e841fea9@oemcomputer> > > "Multiple adjacent string literals (delimited by whitespace), possibly > using > > different quoting conventions, are allowed, and their meaning is the > same as > > their concatenation as long as the newline separating them is not > signifcant to > > the parser." > > I'm not sure it needs clarifying; it's the reference manual after all, > not a tutorial. Right. Over-clarification results in docs that read like the instructions for the holy hand grenade ;-) Raymond From martin at v.loewis.de Sat Nov 27 23:27:12 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat Nov 27 23:27:10 2004 Subject: [Python-Dev] Python 2.4, MS .NET 1.1 and distutils In-Reply-To: <41A89F85.9080703@zope.com> References: <41A7157A.7030405@egenix.com> <41A7597E.2090805@v.loewis.de> <61A404CD-3FC9-11D9-B357-000D93AD379E@mac.com> <653ssgb2.fsf@python.net> <41A762E1.7040404@egenix.com> <41A771F3.3060006@zope.com> <41A7D1B3.7070306@v.loewis.de> <41A89F85.9080703@zope.com> Message-ID: <41A8FF40.4040106@v.loewis.de> Jim Fulton wrote: >> Of course, in that case, it will be sufficient if the other people >> have VC.NET 2003 installed. > > Is that free? If not, that's a big "if". No, it is a product sold by Microsoft. However, many people do have VC.NET installed, despite it not being free. Regards, Martin From ncoghlan at iinet.net.au Sun Nov 28 02:56:49 2004 From: ncoghlan at iinet.net.au (Nick Coghlan) Date: Sun Nov 28 02:56:56 2004 Subject: [Python-Dev] String literal concatenation & docstrings In-Reply-To: <ca471dc2041127130529ecdf98@mail.gmail.com> References: <41A73BE7.7020707@iinet.net.au> <2mpt20veee.fsf@starship.python.net> <41A7496E.9020309@iinet.net.au> <41A78A55.8030802@ocf.berkeley.edu> <864d370904112619532ee559d4@mail.gmail.com> <ca471dc20411270913780b7e63@mail.gmail.com> <41A8D0B2.6060800@ocf.berkeley.edu> <ca471dc2041127130529ecdf98@mail.gmail.com> Message-ID: <41A93061.4080701@iinet.net.au> Guido van Rossum wrote: > I'm not sure it needs clarifying; it's the reference manual after all, > not a tutorial. > > I'd rather let the grammar speak for itself; there's no ambiguity in > that, and the words are just there to clarify the *semantics*. > Also, I'll freely admit that I made the mistake of reading the "string literal concatenation" bit out of context. Reading the preceding section on string literals in general first (which includes the relevant snippet from the grammar), probably would have made it clearer what was going on. http://www.python.org/dev/doc/devel/ref/strings.html Cheers, Nick. I guess there was a reason K&R used ';' after all. . . -- Nick Coghlan | ncoghlan@email.com | Brisbane, Australia --------------------------------------------------------------- http://boredomandlaziness.skystorm.net From anthony at interlink.com.au Sun Nov 28 13:16:54 2004 From: anthony at interlink.com.au (Anthony Baxter) Date: Sun Nov 28 13:17:43 2004 Subject: [Python-Dev] state of 2.4 final release Message-ID: <200411282316.55353.anthony@interlink.com.au> I didn't see any replies to the last post, so I'll ask again with a better subject line - as I said last time, as far as I'm aware, I'm not aware of anyone having done a fix for the issue Tim identified ( http://www.python.org/sf/1069160 ) So, my question is: Is this important enough to delay a 2.4 final for? My plan is currently to release it _this_ _Tuesday_, so I really need an answer soon... I've attached Tim's original message at the end here. At the moment, I'm inclined to say "if it's not fixed, it won't kill us". But that's admittedly my own biases - threading bugs annoy me <wink> I'm happy to defer to more knowlegable types, though - is this so bad that it merits delaying the release? I can't make time to look at it before then - I'm still writing slides for a couple of talks at OSDC. Anthony From: Tim Peters <tim.peters@gmail.com> To: Python Dev <python-dev@python.org> Date: 2004-11-19 13:08 This one is a puzzler. See http://www.python.org/sf/1069160 for details. The short course is that the PyThreadState_SetAsyncExc() implementation fell into a common trap, and can cause segfaults under rare conditions (like every other Python thread segfault bug we've ever had). This is easily repaired (although I've got no interest in doing the coding, or even in contriving a test case -- this was an obvious "by eyeball" bug). The puzzle is how to treat this wrt 2.4. Since it's a critical bug, I suppose it "should" force another release candidate. OTOH, this is a C-only API (there's no exposure from Python) that's never used in the core. We could add code to make it segfault every time <wink>, and nothing in the distribution would notice. OTOH, if we broke its intended behavior while fixing the bug, we'd never know that either. "Never used in the core" means never -- the function isn't tested. On the third hand, it's a simple function with an obvious segfault mode that has an obvious fix. I'll leave it to the release manager <wink>. From aurora00 at gmail.com Sun Nov 28 04:38:58 2004 From: aurora00 at gmail.com (aurora) Date: Sun Nov 28 16:54:15 2004 Subject: [Python-Dev] server modules class diagram Message-ID: <cbd17751041127193872244652@mail.gmail.com> Hi I don't know if I'm sending this to the right mailing list. Anyway I have gone through the Python server module code so many times. I actually come up with a class diagram to help me navigate it. I wonder if this would be useful in offical documentation. Please let me know if this is not the right mailing list for documentations. (The class diagram show be viewed with monospace fonts.) Wai Yip Tung --------------------------------------------------------------------------------------- Standard Python Modules for implementing network servers ...Some description on the collection of classes... There are two important line of classes. The BaseServer and its subclasses handle network connections. The BaseRequestHandler and its subclasses handle the message. ...ForkingMixIn v.s. synchronized... ...mention asyncore, mention cgi... *************************** * The SocketServer module * *************************** ------------------------------- BaseServer ------------------------------- An abstract class that defines the main method handle_request() with pseudo code like below: ------------------------------- def handle_request(): ForkingMixIn req = get_request() ------------------------------- if verify_request(req): Overrides process_request() to try: fork process to handle each process_request(req) request. Handles error in the except: child process. handle_error(req) ------------------------------- close_request(req) def process_request(req): ------------------------------- finish_request(req) ThreadingMixIn close_request(req) ------------------------------- Overrides process_request() to def finish_request(req): spawn new thread to handle each instantiates request. Handles error in the RequestHandlerClass(req) child thread. ------------------------------- Other than handle_request(), all methods may be overridden. ------------------------------- ^ | ------------------------------- TCPServer ------------------------------- ------------------------------- Creates a STREAM socket, then BaseRequestHandler bind() and listen() to it. ------------------------------- Abstract base class that calls Implements get_request() to use setup(), handle() and finish() socket.accept() methods. ------------------------------- ------------------------------- ^ ^ ^ | | | | ---------------------------- ------------------------------- | UDPServer StreamRequestHandler | ---------------------------- ------------------------------- | Creates DGRAM socket. Implements setup() and finish() | to create the rfile/wfile | Overrides get_request() to object from the socket | use socket.recvfrom(). connection. | ---------------------------- ------------------------------- | ^ | | | | | | | ***************************** | | * The BaseHTTPServer module * | | ***************************** | | | | | ------------------------------- ------------------------------- HTTPServer BaseHTTPRequestHandler ------------------------------- ------------------------------- Overrides server_bind to store Implements handle(). Calls the server name. handle_one_request() to parse ------------------------------- HTTP request and invoke the corresponding do_XXX methods. handle_one_request() are called once for each HTTP request in a persistent connection. ------------------------------- ^ | | ************************ | * The SimpleHTTPServer * | ************************ | | | ------------------------------- SimpleHTTPRequestHandler ------------------------------- Implements do_GET() to serve files mapped to URL path. ------------------------------- ^ | | **************************** | * The CGIHTTPServer module * | **************************** | | | ------------------------------- CGIHTTPRequestHandler ------------------------------- Implements do_POST() to find, prepare and invoke CGI programs. ------------------------------- Some observations 1. One exception to the Server / RequestHandler division is the BaseHTTPRequestHandler.handle() method. The request handler is responsible to break down multiple HTTP messages from a persistent TCP connection (See RFC 2616 section 8.1). 2. Strictly speaking UDPServer should not be a specialization of TCPServer. 3. The BaseServer.finish_request() is a bit of a misnomer. From there it instantiates a RequestHandler where most application logic actually begins. 4. Note that BaseRequestHandler put most application logic in the __init__ method. (Unfortunately this does not give the caller much fine control. Also when an exception is thrown in __init__, the object reference is not available at all). Also notice that finish() is not in a finally clause and may not be called when an exception is thrown. From mwh at python.net Sun Nov 28 18:20:18 2004 From: mwh at python.net (Michael Hudson) Date: Sun Nov 28 18:20:28 2004 Subject: [Python-Dev] String literal concatenation & docstrings In-Reply-To: <001001c4d4c5$7d2d8ec0$e841fea9@oemcomputer> (Raymond Hettinger's message of "Sat, 27 Nov 2004 16:10:14 -0500") References: <001001c4d4c5$7d2d8ec0$e841fea9@oemcomputer> Message-ID: <2msm6tubzh.fsf@starship.python.net> "Raymond Hettinger" <python@rcn.com> writes: > Right. Over-clarification results in docs that read like the > instructions for the holy hand grenade ;-) Well said. Cheers, mwh -- Famous remarks are very seldom quoted correctly. -- Simeon Strunsky From martin at v.loewis.de Sun Nov 28 23:27:13 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun Nov 28 23:27:11 2004 Subject: [Python-Dev] state of 2.4 final release In-Reply-To: <200411282316.55353.anthony@interlink.com.au> References: <200411282316.55353.anthony@interlink.com.au> Message-ID: <41AA50C1.4010102@v.loewis.de> Anthony Baxter wrote: > So, my question is: Is this important enough to delay a 2.4 final for? I think we should release the code with that bug. I feel that this async exceptions thing has many more problems in it, and we may have to revert the entire feature. Regards, Martin From gvanrossum at gmail.com Mon Nov 29 01:55:54 2004 From: gvanrossum at gmail.com (Guido van Rossum) Date: Mon Nov 29 01:55:58 2004 Subject: [Python-Dev] String literal concatenation & docstrings In-Reply-To: <2msm6tubzh.fsf@starship.python.net> References: <001001c4d4c5$7d2d8ec0$e841fea9@oemcomputer> <2msm6tubzh.fsf@starship.python.net> Message-ID: <ca471dc2041128165530532ab4@mail.gmail.com> > > Right. Over-clarification results in docs that read like the > > instructions for the holy hand grenade ;-) > > Well said. Except that now I can't find the adjacent string literals in the grammar any more! I'm looking al http://www.python.org/dev/doc/devel/ref/grammar.txt The path goes from primary to atom to literal to stringliteral (and from there on into lexical detail) and nowhere does the grammar show that multiple string literals are allowed. Adding a single + after stringliteral in the expansion for literal would fix this. Once that is fixed, we could probably reduce the text of the offending section somewhat to use the phrase "where allowed by the grammar" and skip the mentioning of different quoting conventions or intervening whitespace. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From anthony at interlink.com.au Mon Nov 29 02:39:20 2004 From: anthony at interlink.com.au (Anthony Baxter) Date: Mon Nov 29 02:40:48 2004 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Lib/email __init__.py, 1.34, 1.35 In-Reply-To: <E1CYa3V-0004xH-OH@sc8-pr-cvs1.sourceforge.net> References: <E1CYa3V-0004xH-OH@sc8-pr-cvs1.sourceforge.net> Message-ID: <200411291239.20848.anthony@interlink.com.au> On Monday 29 November 2004 12:10, bwarsaw@users.sourceforge.net wrote: > Update of /cvsroot/python/python/dist/src/Lib/email > In directory sc8-pr-cvs1.sourceforge.net:/tmp/cvs-serv19040 > > Modified Files: > __init__.py > Log Message: > There's likely nothing more to do to the email package before Python 2.4 is > final, so I'm marking email's version number as "3.0" (e.g. final). Are there any other packages that have their own version numbers that should be updated? From anthony at interlink.com.au Mon Nov 29 02:50:18 2004 From: anthony at interlink.com.au (Anthony Baxter) Date: Mon Nov 29 03:24:07 2004 Subject: [Python-Dev] TRUNK FREEZE for 2.4 final from 2100 UTC, 29-11-2004 Message-ID: <200411291250.18764.anthony@interlink.com.au> Ok, we're about ready for the 2.4 final release. Please hold off any checkins post 21:00 UTC (so in about 19-20 hours from now). Thanks, Anthony From bac at OCF.Berkeley.EDU Mon Nov 29 03:51:27 2004 From: bac at OCF.Berkeley.EDU (Brett C.) Date: Mon Nov 29 03:51:30 2004 Subject: [Python-Dev] String literal concatenation & docstrings In-Reply-To: <ca471dc2041128165530532ab4@mail.gmail.com> References: <001001c4d4c5$7d2d8ec0$e841fea9@oemcomputer> <2msm6tubzh.fsf@starship.python.net> <ca471dc2041128165530532ab4@mail.gmail.com> Message-ID: <41AA8EAF.1050600@ocf.berkeley.edu> Guido van Rossum wrote: >>>Right. Over-clarification results in docs that read like the >>>instructions for the holy hand grenade ;-) >> >>Well said. > > > Except that now I can't find the adjacent string literals in the > grammar any more! > > I'm looking al http://www.python.org/dev/doc/devel/ref/grammar.txt > > The path goes from primary to atom to literal to stringliteral (and > from there on into lexical detail) and nowhere does the grammar show > that multiple string literals are allowed. Adding a single + after > stringliteral in the expansion for literal would fix this [SNIP] But if you look at Grammar/Grammar you will notice that atom goes to STRING+ which should cover this. Is that grammar.txt file generated from Grammar/Grammar or is it done by hand? -Brett From gvanrossum at gmail.com Mon Nov 29 05:32:29 2004 From: gvanrossum at gmail.com (Guido van Rossum) Date: Mon Nov 29 05:32:32 2004 Subject: [Python-Dev] String literal concatenation & docstrings In-Reply-To: <41AA8EAF.1050600@ocf.berkeley.edu> References: <001001c4d4c5$7d2d8ec0$e841fea9@oemcomputer> <2msm6tubzh.fsf@starship.python.net> <ca471dc2041128165530532ab4@mail.gmail.com> <41AA8EAF.1050600@ocf.berkeley.edu> Message-ID: <ca471dc204112820321dee1a2a@mail.gmail.com> > But if you look at Grammar/Grammar you will notice that atom goes to STRING+ > which should cover this. Of course, otherwise it wouldn't work! :-) > Is that grammar.txt file generated from > Grammar/Grammar or is it done by hand? By hand. The reference manual has more detail (Grammar/Grammar doesn't say anything about what a STRING is, or other literals) and presents some rules in a more readable form which wouldn't work for Grammar/Grammar given the LL1 constraints of pgen. It also chooses more readable names. But the translation process is fallible. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From aahz at pythoncraft.com Mon Nov 29 06:01:14 2004 From: aahz at pythoncraft.com (Aahz) Date: Mon Nov 29 06:01:17 2004 Subject: [Python-Dev] server modules class diagram In-Reply-To: <cbd17751041127193872244652@mail.gmail.com> References: <cbd17751041127193872244652@mail.gmail.com> Message-ID: <20041129050114.GA4293@panix.com> On Sat, Nov 27, 2004, aurora wrote: > > Hi I don't know if I'm sending this to the right mailing list. Anyway > I have gone through the Python server module code so many times. I > actually come up with a class diagram to help me navigate it. I wonder > if this would be useful in offical documentation. Please let me know > if this is not the right mailing list for documentations. (The class > diagram show be viewed with monospace fonts.) If you look at the Python docs, the bottom of every page says, "See About the Python Documentation for information on suggesting changes". -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ WiFi is the SCSI of the 21st Century -- there are fundamental technical reasons for sacrificing a goat. (with no apologies to John Woods) From anthony at interlink.com.au Mon Nov 29 10:03:58 2004 From: anthony at interlink.com.au (Anthony Baxter) Date: Mon Nov 29 10:05:29 2004 Subject: [Python-Dev] TRUNK FREEZE for 2.4 final from 2100 UTC, 29-11-2004 In-Reply-To: <200411291250.18764.anthony@interlink.com.au> References: <200411291250.18764.anthony@interlink.com.au> Message-ID: <200411292004.00001.anthony@interlink.com.au> On Monday 29 November 2004 12:50, Anthony Baxter wrote: > Ok, we're about ready for the 2.4 final release. Please hold off any > checkins post 21:00 UTC (so in about 19-20 hours from now). I should also note that shortly after the release is done and we've confirmed that there's no brown-paper-bag bugs, I'll be branching the release24-maint branch in CVS. Anthony From niemeyer at conectiva.com Mon Nov 29 20:04:48 2004 From: niemeyer at conectiva.com (Gustavo Niemeyer) Date: Tue Nov 30 02:36:46 2004 Subject: [Python-Dev] File encodings Message-ID: <20041129190448.GA23399@burma.localdomain> Greetings, Today, while trying to internationalize a program I'm working on, I found an interesting side-effect of how we're dealing with encoding of unicode strings while being written to files. Suppose the following example: # -*- encoding: iso-8859-1 -*- print u"?" This will correctly print the string '?', as expected. Now, what surprises me, is that the following code won't work in an equivalent way (unless using sys.setdefaultencoding()): # -*- encoding: iso-8859-1 -*- import sys sys.stdout.write(u"?\n") This will raise the following error: Traceback (most recent call last): File "asd.py", line 3, in ? sys.stdout.write(u"?") UnicodeEncodeError: 'ascii' codec can't encode character u'\xe1' in position 0:ordinal not in range(128) This difference may become a really annoying problem when trying to internationalize programs, since it's usual to see third-party code dealing with sys.stdout, instead of using 'print'. The standard optparse module, for instance, has a reference to sys.stdout which is used in the default --help handling mechanism. Given the fact that files have an 'encoding' parameter, and that any unicode strings with characters not in the 0-127 range will raise an exception if being written to files, isn't it reasonable to respect the 'encoding' attribute whenever writing data to a file? The workaround for that problem is to either use the evil-considered sys.setdefaultencoding(), or to wrap sys.stdout. IMO, both options seem unreasonable for such a common idiom. -- Gustavo Niemeyer http://niemeyer.net From tim.peters at gmail.com Tue Nov 30 04:18:40 2004 From: tim.peters at gmail.com (Tim Peters) Date: Tue Nov 30 04:19:30 2004 Subject: [Python-Dev] state of 2.4 final release In-Reply-To: <200411282316.55353.anthony@interlink.com.au> References: <200411282316.55353.anthony@interlink.com.au> Message-ID: <1f7befae04112919183144973b@mail.gmail.com> [Anthony Baxter] > I didn't see any replies to the last post, so I'll ask again with a > better subject line - as I said last time, as far as I'm aware, I'm > not aware of anyone having done a fix for the issue Tim identified > ( http://www.python.org/sf/1069160 ) > > So, my question is: Is this important enough to delay a 2.4 final > for? Not according to me; said before I'd be happy if everyone pretended I hadn't filed that report until a month after 2.4 final was released. From bob at redivi.com Tue Nov 30 04:45:26 2004 From: bob at redivi.com (Bob Ippolito) Date: Tue Nov 30 04:45:37 2004 Subject: [Python-Dev] File encodings In-Reply-To: <20041129190448.GA23399@burma.localdomain> References: <20041129190448.GA23399@burma.localdomain> Message-ID: <456E4FEC-4282-11D9-A720-000A9567635C@redivi.com> On Nov 29, 2004, at 2:04 PM, Gustavo Niemeyer wrote: > Today, while trying to internationalize a program I'm working on, > I found an interesting side-effect of how we're dealing with > encoding of unicode strings while being written to files. > > Suppose the following example: > > # -*- encoding: iso-8859-1 -*- > print u"?" > > This will correctly print the string '?', as expected. Now, what > surprises me, is that the following code won't work in an equivalent > way (unless using sys.setdefaultencoding()): That doesn't work here, where sys.getdefaultencoding() is 'ascii', as expected. > # -*- encoding: iso-8859-1 -*- > import sys > sys.stdout.write(u"?\n") > > This will raise the following error: > > Traceback (most recent call last): > File "asd.py", line 3, in ? > sys.stdout.write(u"?") > UnicodeEncodeError: 'ascii' codec can't encode character u'\xe1' > in position 0:ordinal not in range(128) That's expected. > This difference may become a really annoying problem when trying to > internationalize programs, since it's usual to see third-party code > dealing with sys.stdout, instead of using 'print'. The standard > optparse module, for instance, has a reference to sys.stdout which > is used in the default --help handling mechanism. > > Given the fact that files have an 'encoding' parameter, and that > any unicode strings with characters not in the 0-127 range will > raise an exception if being written to files, isn't it reasonable > to respect the 'encoding' attribute whenever writing data to a > file? No, because you don't know it's a file. You're calling a function with a unicode object. The function doesn't know that the object was some unicode object that came from a source file of some particular encoding. > The workaround for that problem is to either use the evil-considered > sys.setdefaultencoding(), or to wrap sys.stdout. IMO, both options > seem unreasonable for such a common idiom. There's no guaranteed correlation whatsoever between the claimed encoding of your source document and the encoding of the user's terminal, why do you want there to be? What if you have some source files with 'foo' encoding and others with 'bar' encoding? What about ascii encoded source documents that use escape sequences to represent non-ascii characters? What you want doesn't make any sense so long as python strings and file objects deal in bytes not characters :) Wrapping sys.stdout is the ONLY reasonable solution. This is the idiom that I use. It's painless and works quite well: import sys import codecs sys.stdout = codecs.getwriter('utf-8')(sys.stdout) -bob From martin at v.loewis.de Tue Nov 30 09:44:05 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue Nov 30 09:44:10 2004 Subject: [Python-Dev] File encodings In-Reply-To: <20041129190448.GA23399@burma.localdomain> References: <20041129190448.GA23399@burma.localdomain> Message-ID: <41AC32D5.5000707@v.loewis.de> Gustavo Niemeyer wrote: > Given the fact that files have an 'encoding' parameter, and that > any unicode strings with characters not in the 0-127 range will > raise an exception if being written to files, isn't it reasonable > to respect the 'encoding' attribute whenever writing data to a > file? In general, files don't have an encoding parameter - sys.stdout is an exception. The reason why this works for print and not for write is that I considered "print unicodeobject" important, and wanted to implement that. file.write is an entirely different code path, so it doesn't currently consider Unicode objects; instead, it only supports strings (or, more generally, buffers). > This difference may become a really annoying problem when trying to > internationalize programs, since it's usual to see third-party code > dealing with sys.stdout, instead of using 'print'. Apparently, it isn't important enough that somebody had analysed this, and offered a patch. In any case, it would be quite unreliable to pass unicode strings to .write even *if* .write supported .encoding, since most files don't have .encoding. Even sys.stdout does not always have .encoding - only when it is a terminal, and only if we managed to find out what the encoding of the terminal is. Regards, Martin From mal at egenix.com Tue Nov 30 09:52:34 2004 From: mal at egenix.com (M.-A. Lemburg) Date: Tue Nov 30 09:52:28 2004 Subject: [Python-Dev] File encodings In-Reply-To: <20041129190448.GA23399@burma.localdomain> References: <20041129190448.GA23399@burma.localdomain> Message-ID: <41AC34D2.1060909@egenix.com> Gustavo Niemeyer wrote: > Greetings, > > Today, while trying to internationalize a program I'm working on, > I found an interesting side-effect of how we're dealing with > encoding of unicode strings while being written to files. > > Suppose the following example: > > # -*- encoding: iso-8859-1 -*- > print u"?" > > This will correctly print the string '?', as expected. Now, what > surprises me, is that the following code won't work in an equivalent > way (unless using sys.setdefaultencoding()): > > # -*- encoding: iso-8859-1 -*- > import sys > sys.stdout.write(u"?\n") > > This will raise the following error: > > Traceback (most recent call last): > File "asd.py", line 3, in ? > sys.stdout.write(u"?") > UnicodeEncodeError: 'ascii' codec can't encode character u'\xe1' > in position 0:ordinal not in range(128) > > This difference may become a really annoying problem when trying to > internationalize programs, since it's usual to see third-party code > dealing with sys.stdout, instead of using 'print'. The standard > optparse module, for instance, has a reference to sys.stdout which > is used in the default --help handling mechanism. You are mixing things here: The source encoding is meant for the parser and defines the way Unicode literals are converted into Unicode objects. The encoding used on the stdout stream doesn't have anything to do with the source code encoding and has to be handled differently. The idiom presented by Bob is the right way to go: wrap sys.stdout with a StreamEncoder. Using sys.setdefaultencoding() is *not* the right solution to the problem. In general when writing programs that are targetted for i18n, you should use Unicode for all text data and convert from Unicode to 8-bit only at the IO/UI layer. The various wrappers in the codecs module make this rather easy. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Nov 30 2004) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From anthony at interlink.com.au Tue Nov 30 13:31:34 2004 From: anthony at interlink.com.au (Anthony Baxter) Date: Tue Nov 30 13:33:02 2004 Subject: [Python-Dev] RELEASED Python 2.4 (final) Message-ID: <200411302331.43246.anthony@interlink.com.au> On behalf of the Python development team and the Python community, I'm happy to announce the release of Python 2.4. Python 2.4 is a final, stable release, and we can recommend that Python users upgrade to this version. Python 2.4 is the result of almost 18 month's worth of work on top of Python 2.3 and represents another stage in the careful evolution of Python. New language features have been kept to a minimum, many bugs have been fixed and a wide variety of improvements have been made. Notable changes in Python 2.4 include improvements to the importing of modules, generator expressions, function decorators, a number of new modules (including subprocess, decimal and cookielib) and countless numbers of fixed bugs and smaller enhancements. For more, see the (subjective) highlights, the release notes, or Andrew Kuchling's What's New In Python, all available from the 2.4 web page. http://www.python.org/2.4/ Please log any problems you have with this release in the SourceForge bug tracker (noting that you're using Python 2.4): http://sourceforge.net/bugs/?group_id=5470 Enjoy the new (stable!) release, Anthony Anthony Baxter anthony@python.org Python Release Manager (on behalf of the entire python-dev team) -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://mail.python.org/pipermail/python-dev/attachments/20041130/13167337/attachment.pgp From niemeyer at conectiva.com Tue Nov 30 13:20:13 2004 From: niemeyer at conectiva.com (Gustavo Niemeyer) Date: Tue Nov 30 14:20:50 2004 Subject: [Python-Dev] File encodings In-Reply-To: <456E4FEC-4282-11D9-A720-000A9567635C@redivi.com> References: <20041129190448.GA23399@burma.localdomain> <456E4FEC-4282-11D9-A720-000A9567635C@redivi.com> Message-ID: <20041130122013.GA6610@burma.localdomain> Hello Bob, [...] > >Given the fact that files have an 'encoding' parameter, and that > >any unicode strings with characters not in the 0-127 range will > >raise an exception if being written to files, isn't it reasonable > >to respect the 'encoding' attribute whenever writing data to a > >file? > > No, because you don't know it's a file. You're calling a function with > a unicode object. The function doesn't know that the object was some > unicode object that came from a source file of some particular > encoding. I don't understand what you're saying here. The file knows itself is a file. The write function knows the parameter is unicode. > >The workaround for that problem is to either use the evil-considered > >sys.setdefaultencoding(), or to wrap sys.stdout. IMO, both options > >seem unreasonable for such a common idiom. > > There's no guaranteed correlation whatsoever between the claimed > encoding of your source document and the encoding of the user's > terminal, why do you want there to be? What if you have some source I don't. I want the write() function of file objects to respect the encoding attribute of these objects. This is already being done when print is used. I'm proposing to extend that behavior to the write function. That's all. > files with 'foo' encoding and others with 'bar' encoding? What about > ascii encoded source documents that use escape sequences to represent > non-ascii characters? What you want doesn't make any sense so long as > python strings and file objects deal in bytes not characters :) Please, take a long breath, and read my message again. :-) > Wrapping sys.stdout is the ONLY reasonable solution. [...] No, it's not. But I'm glad to know other people is also doing workarounds for that problem. -- Gustavo Niemeyer http://niemeyer.net From anthony at interlink.com.au Tue Nov 30 14:23:17 2004 From: anthony at interlink.com.au (Anthony Baxter) Date: Tue Nov 30 14:24:40 2004 Subject: [Python-Dev] TRUNK UNFROZEN; release24-maint branch has been cut Message-ID: <200412010023.17728.anthony@interlink.com.au> I've cut the release24-maint branch, and updated the Include/patchlevel.h on trunk and branch (trunk is now 2.5a0, branch is 2.4+) The trunk and the branch are now both unfrozen and suitable for checkins. The feature freeze on the trunk is lifted. Remember - if you're checking bugfixes into the trunk, either backport them to the branch, or else mark the commit message with 'bugfix candidate' or 'backport candidate' or the like. Next up will be a 2.3.5 release. I'm going to be travelling for a large chunk of December (at very short notice) so it's likely that this will happen at the start of January. If someone else wants to cut a 2.3.5 sooner than that, please feel free to volunteer! 2.3.5 will be the last 2.3.x release, barring some almighty cockup - the next scheduled release will be 2.4.1, which will probably happen around May 2005. Anthony and yes, I'm drinking. From niemeyer at conectiva.com Tue Nov 30 13:29:08 2004 From: niemeyer at conectiva.com (Gustavo Niemeyer) Date: Tue Nov 30 14:29:48 2004 Subject: [Python-Dev] File encodings In-Reply-To: <41AC32D5.5000707@v.loewis.de> References: <20041129190448.GA23399@burma.localdomain> <41AC32D5.5000707@v.loewis.de> Message-ID: <20041130122908.GB6610@burma.localdomain> > Gustavo Niemeyer wrote: > >Given the fact that files have an 'encoding' parameter, and that > >any unicode strings with characters not in the 0-127 range will > >raise an exception if being written to files, isn't it reasonable > >to respect the 'encoding' attribute whenever writing data to a > >file? > > In general, files don't have an encoding parameter - sys.stdout > is an exception. That's the only case I'd like to solve. If there are platforms that don't know how to set it, we could make the encoding attribute writable, and that would allow people to easily set it to the encoding which is deemed correct in their systems. > The reason why this works for print and not for write is that > I considered "print unicodeobject" important, and wanted to > implement that. file.write is an entirely different code path, > so it doesn't currently consider Unicode objects; instead, it > only supports strings (or, more generally, buffers). I understand your reasoning behind it, and would like to extend your idea to the write function, allowing anyone to use the common sys.stdout idiom to implement print-like functionality (like optparse and many others). For normal files, the absence of the encoding parameter would ensure the current behavior. > > This difference may become a really annoying problem when trying to > > internationalize programs, since it's usual to see third-party code > > dealing with sys.stdout, instead of using 'print'. > > Apparently, it isn't important enough that somebody had analysed this, > and offered a patch. In any case, it would be quite unreliable to That's what I'm doing here! :-) > pass unicode strings to .write even *if* .write supported .encoding, > since most files don't have .encoding. Even sys.stdout does not always > have .encoding - only when it is a terminal, and only if we managed to > find out what the encoding of the terminal is. I think that's acceptable. The encoding parameter is meant for output streams, and Python does its best to try to find a reasonable value for showing output strings. Thanks for your answer and clarifications, -- Gustavo Niemeyer http://niemeyer.net From niemeyer at conectiva.com Tue Nov 30 14:09:34 2004 From: niemeyer at conectiva.com (Gustavo Niemeyer) Date: Tue Nov 30 15:10:40 2004 Subject: [Python-Dev] File encodings In-Reply-To: <41AC34D2.1060909@egenix.com> References: <20041129190448.GA23399@burma.localdomain> <41AC34D2.1060909@egenix.com> Message-ID: <20041130130934.GC6610@burma.localdomain> [...] > You are mixing things here: > > The source encoding is meant for the > parser and defines the way Unicode literals are converted > into Unicode objects. > > The encoding used on the stdout stream doesn't have anything > to do with the source code encoding and has to be handled > differently. Sorry. I probably wasn't clear enough in my message. I understand the issue, and I'm not discussing source encoding at all. The only problem I'd like to solve is that of output streams not being able to have unicode strings written. > The idiom presented by Bob is the right way to go: wrap > sys.stdout with a StreamEncoder. I don't see that as a good solution, since every Python software that is internationalizaed will have do figure out this wrapping, introducing extra overhead unnecessarily. > Using sys.setdefaultencoding() is *not* the right solution > to the problem. I understand. > In general when writing programs that are targetted for > i18n, you should use Unicode for all text data and > convert from Unicode to 8-bit only at the IO/UI layer. That's what I think as well. I just would expect that Python was kind enough to allow me to tell which output encoding I want, instead of wrapping the sys.stdout object with a non-native-file. IOW, being widely necessary, handling internationalization without wrapping sys.stdout everytime seems like a good step for a language like Python. > The various wrappers in the codecs module make this > rather easy. Thanks for the suggestion! -- Gustavo Niemeyer http://niemeyer.net From gvanrossum at gmail.com Tue Nov 30 18:04:33 2004 From: gvanrossum at gmail.com (Guido van Rossum) Date: Tue Nov 30 18:05:17 2004 Subject: [Python-Dev] RELEASED Python 2.4 (final) In-Reply-To: <200411302331.43246.anthony@interlink.com.au> References: <200411302331.43246.anthony@interlink.com.au> Message-ID: <ca471dc204113009041ce40ac4@mail.gmail.com> On Tue, 30 Nov 2004 23:31:34 +1100, Anthony Baxter <anthony@interlink.com.au> wrote: > On behalf of the Python development team and the Python community, I'm > happy to announce the release of Python 2.4. Anthony, congratulations with this smooth release! Thanks for all the hard work. Enjoy your beer, I'm celebrating with you in spirit. Thanks also to all the other developers who contributed to this release, and to the many thousands of users who helped with the the beta testing. Let's hope for record downloads, -- --Guido van Rossum (home page: http://www.python.org/~guido/) From walter at livinglogic.de Tue Nov 30 18:43:37 2004 From: walter at livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=) Date: Tue Nov 30 18:43:40 2004 Subject: [Python-Dev] File encodings In-Reply-To: <20041130130934.GC6610@burma.localdomain> References: <20041129190448.GA23399@burma.localdomain> <41AC34D2.1060909@egenix.com> <20041130130934.GC6610@burma.localdomain> Message-ID: <41ACB149.6050300@livinglogic.de> Gustavo Niemeyer wrote: > [...] > >>You are mixing things here: >> >>The source encoding is meant for the >>parser and defines the way Unicode literals are converted >>into Unicode objects. >> >>The encoding used on the stdout stream doesn't have anything >>to do with the source code encoding and has to be handled >>differently. > > Sorry. I probably wasn't clear enough in my message. I understand > the issue, and I'm not discussing source encoding at all. The > only problem I'd like to solve is that of output streams not > being able to have unicode strings written. > >>The idiom presented by Bob is the right way to go: wrap >>sys.stdout with a StreamEncoder. > > I don't see that as a good solution, since every Python software > that is internationalizaed will have do figure out this wrapping, > introducing extra overhead unnecessarily. This wrapping is probably necessary for stateful encodings. If you had a sys.stdout.encoding=="utf-16", print would probably add the BOM every time a unicode object is printed. This doesn't happen if you wrap sys.stdout in a StreamWriter. > [...] > That's what I think as well. I just would expect that Python was > kind enough to allow me to tell which output encoding I want, > instead of wrapping the sys.stdout object with a non-native-file. > > IOW, being widely necessary, handling internationalization without > wrapping sys.stdout everytime seems like a good step for a language > like Python. You can't have stateful encodings without something that keeps state. The only thing that does keep state in Python is a StreamReader/StreamWriter. Bye, Walter D?rwald From ark at acm.org Tue Nov 30 18:49:38 2004 From: ark at acm.org (Andrew Koenig) Date: Tue Nov 30 18:49:42 2004 Subject: [Python-Dev] Trouble installing 2.4 Message-ID: <000001c4d704$f6d4f8d0$6402a8c0@arkdesktop> I'm using Windows XP SP2. Uninstalled 2.3, installed 2.4 (running as me, not as administrator). No problems so far. Tried installing pywin32-203.win32-py2.4.exe When I try to install it as me, it gets as far as "ready to install." When I click Next, it says Can't load Python for pre-install script and quits, even though earlier it said it had found Python 2.4 in the registry. When I try to install it as Administrator, it quits immediately, saying that it couldn't locate a Python 2.4 installation. My hypothesis: When I install 2.4 as me, it puts it in my user registry, not the system-wide registry, and then pywin32 can't find it. I'm going to unstall and try again as Administrator. From niemeyer at conectiva.com Tue Nov 30 18:11:53 2004 From: niemeyer at conectiva.com (Gustavo Niemeyer) Date: Tue Nov 30 19:12:28 2004 Subject: [Python-Dev] File encodings In-Reply-To: <41ACB149.6050300@livinglogic.de> References: <20041129190448.GA23399@burma.localdomain> <41AC34D2.1060909@egenix.com> <20041130130934.GC6610@burma.localdomain> <41ACB149.6050300@livinglogic.de> Message-ID: <20041130171153.GA8780@burma.localdomain> Hello Walter, > >I don't see that as a good solution, since every Python software > >that is internationalizaed will have do figure out this wrapping, > >introducing extra overhead unnecessarily. > > This wrapping is probably necessary for stateful encodings. If you > had a sys.stdout.encoding=="utf-16", print would probably add the > BOM every time a unicode object is printed. This doesn't happen if > you wrap sys.stdout in a StreamWriter. I'm not sure this is an issue for a terminal output stream, which is the case I'm trying to find a solution for. Otherwise, Python would already be in trouble for using this scheme in the print statement. Can you show an example of the print statement not working? -- Gustavo Niemeyer http://niemeyer.net From ark-mlist at att.net Tue Nov 30 19:16:34 2004 From: ark-mlist at att.net (Andrew Koenig) Date: Tue Nov 30 19:16:36 2004 Subject: [Python-Dev] Trouble installing 2.4 In-Reply-To: <000001c4d704$f6d4f8d0$6402a8c0@arkdesktop> Message-ID: <000401c4d708$ba149dc0$6402a8c0@arkdesktop> Follow-up: When I install Python as Administrator, all is well. In that case (but not when installing it as me), it asks whether I want to install it for all users or for myself only. I then install pywin32 and it works. So it may be that a caveat is in order to people who do not install 2.4 as Administrator. From walter at livinglogic.de Tue Nov 30 20:26:36 2004 From: walter at livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=) Date: Tue Nov 30 20:26:40 2004 Subject: [Python-Dev] File encodings In-Reply-To: <20041130171153.GA8780@burma.localdomain> References: <20041129190448.GA23399@burma.localdomain> <41AC34D2.1060909@egenix.com> <20041130130934.GC6610@burma.localdomain> <41ACB149.6050300@livinglogic.de> <20041130171153.GA8780@burma.localdomain> Message-ID: <41ACC96C.5030102@livinglogic.de> Gustavo Niemeyer wrote: > Hello Walter, > >>>I don't see that as a good solution, since every Python software >>>that is internationalizaed will have do figure out this wrapping, >>>introducing extra overhead unnecessarily. >> >>This wrapping is probably necessary for stateful encodings. If you >>had a sys.stdout.encoding=="utf-16", print would probably add the >>BOM every time a unicode object is printed. This doesn't happen if >>you wrap sys.stdout in a StreamWriter. > > I'm not sure this is an issue for a terminal output stream, which > is the case I'm trying to find a solution for. Otherwise, Python > would already be in trouble for using this scheme in the print > statement. Can you show an example of the print statement not > working? No, I can't. Python doesn't accept UTF-16 as encoding. This works: > LANG=de_DE.UTF-8 python2.4 Python 2.4 (#1, Nov 30 2004, 14:16:24) [GCC 2.96 20000731 (Red Hat Linux 7.3 2.96-113)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import sys >>> sys.stdout.encoding 'UTF-8' This doesn't: > LANG=de_DE.UTF-16 python2.4 Python 2.4 (#1, Nov 30 2004, 14:16:24) [GCC 2.96 20000731 (Red Hat Linux 7.3 2.96-113)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import sys >>> sys.stdout.encoding 'ANSI_X3.4-1968' Bye, Walter D?rwald From theller at python.net Tue Nov 30 20:46:51 2004 From: theller at python.net (Thomas Heller) Date: Tue Nov 30 20:45:58 2004 Subject: [Python-Dev] Python.org current docs Message-ID: <8y8jrufo.fsf@python.net> http://www.python.org/doc/current/ and http://docs.python.org/ still point to 2.3.4 docs. Thomas From astrand at lysator.liu.se Tue Nov 30 21:33:06 2004 From: astrand at lysator.liu.se (=?iso-8859-1?Q?Peter_=C5strand?=) Date: Tue Nov 30 21:33:14 2004 Subject: [Python-Dev] Small subprocess patch Message-ID: <Pine.GSO.4.51L2.0411302123220.18754@koeberg.lysator.liu.se> I'm planning to change the signature for subprocess.call slightly: -def call(*args, **kwargs): +def call(*popenargs, **kwargs): The purpose is to make it clearer that "args" in this context is not the same as the "args" argument to the Popen constructor. Two questions: 1) Is it OK to commit changes like this on the 2.4 branch, in addition to trunk? 2) Anyone that thinks that "kwargs" should be changed into "popenkwargs"? /Peter ?strand <astrand@lysator.liu.se> From martin at v.loewis.de Tue Nov 30 22:06:28 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue Nov 30 22:06:30 2004 Subject: [Python-Dev] Trouble installing 2.4 In-Reply-To: <000401c4d708$ba149dc0$6402a8c0@arkdesktop> References: <000401c4d708$ba149dc0$6402a8c0@arkdesktop> Message-ID: <41ACE0D4.3090507@v.loewis.de> Andrew Koenig wrote: > So it may be that a caveat is in order to people who do not install 2.4 as > Administrator. I think the trouble is not with 2.4, here - the trouble is with installing pywin32. As you said, the installation of Python itself went fine. > My hypothesis: When I install 2.4 as me, it puts it in my user > registry, not the system-wide registry, I can confirm this hypothesis. In a per-user installation, the registry settings are deliberately change for the user, not for the entire system. Otherwise, it wouldn't be per-user. Also, the user might not be able to write to the machine registry (unless he is a member of the Power Users group). > and then pywin32 can't find it. That sounds likely, but I cannot confirm it. If it is, it is a bug in pywin32 (and, in turn, possibly in distutils). Regards, Martin From fdrake at acm.org Tue Nov 30 22:12:37 2004 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Tue Nov 30 22:12:59 2004 Subject: [Python-Dev] Python.org current docs In-Reply-To: <8y8jrufo.fsf@python.net> References: <8y8jrufo.fsf@python.net> Message-ID: <200411301612.37180.fdrake@acm.org> On Tuesday 30 November 2004 02:46 pm, Thomas Heller wrote: > http://www.python.org/doc/current/ > and > http://docs.python.org/ > > still point to 2.3.4 docs. I'll be fixing that up tonight. -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> From astrand at lysator.liu.se Tue Nov 30 23:00:20 2004 From: astrand at lysator.liu.se (Peter Astrand) Date: Tue Nov 30 23:00:27 2004 Subject: [Python-Dev] Re: Small subprocess patch In-Reply-To: <Pine.GSO.4.51L2.0411302123220.18754@koeberg.lysator.liu.se> References: <Pine.GSO.4.51L2.0411302123220.18754@koeberg.lysator.liu.se> Message-ID: <Pine.GSO.4.51L2.0411302255440.21491@koeberg.lysator.liu.se> On Tue, 30 Nov 2004, Peter ?strand wrote: > 1) Is it OK to commit changes like this on the 2.4 branch, in addition to > trunk? I'm also wondering if patch 1071755 and 1071764 should go into release24-maint: * 1071755 makes subprocess raise TypeError if Popen is called with a bufsize that is not an integer. * 1071764 adds a new, small utility function. /Peter ?strand <astrand@lysator.liu.se>