From fdrake@acm.org Mon Oct 1 15:17:06 2001 From: fdrake@acm.org (Fred L. Drake, Jr.) Date: Mon, 1 Oct 2001 10:17:06 -0400 Subject: [Python-Dev] Integrating Expat In-Reply-To: <200109301453.QAA21436@paros.informatik.hu-berlin.de> References: <200109301453.QAA21436@paros.informatik.hu-berlin.de> <3BB74B99.3B230398@lemburg.com> <200109301708.TAA22863@paros.informatik.hu-berlin.de> Message-ID: <15288.31458.735886.325225@grendel.zope.com> Martin von Loewis writes: > [I know I've asked this before, but Fred wanted me to ask it again :-] Actually, I think I simply suggested the forum so that others could comment as well. ;-) > What do you think about an integration of Expat into Python, to be > always able to build pyexpat (and with the same version also)? > Which version of Expat would you use? Would you put the expat files > into a separate directory, or all into modules? I have mixed feelings. There are really two things that we could do: We could add Expat to our CVS repository, which means syncing a bunch of files everytime a new Expat release comes out, or we could bundle the Expat sources with the Python source distribution when the distribution is built, but not add them to CVS. This avoids the extra files in CVS, but complicates construction of the distribution and adds a new wrinkle to the configuration management. > Here is my proposal: Integrate Expat 2.95.2 for release together with > Python 2.2; into an expat subdirectory of Modules (taking only the lib > files of expat). > > This would affect build procedures on all targets; in particular, > pyexpat would not link to a shared expat DLL, but incorporate the > object files. For the "Parsed XML" Zope product, we included the sources for the Expat library in our CVS, but added our own configure.in and other build-control files, which are simpler than those included with Expat (since it only needs to build the static library). This seems to work reasonably well, and does not introduce new wrinkles to the configuration management. So I think we agree on the approach to take. M.-A. Lemburg writes: > Are you sure that we should choose expat as "native" XML parser ? > > There are other candidates which would fit this role just > as well (in particular, Fredrik's sgmlop looks like a nice > extension since it not only works with XML but also many > other meta languages). See Martin's comments about this. I think this precludes inclusion of sgmlop until the problems it has have been addressed in the implementation. I'm not sure what "meta languages" it handles; I thought it only dealt with XML/XHTML and HTML document markup. > If you want a very fast validating XML parser, RXP would also > be a good choice -- AFAIK, the RXP folks would allow us to > ship RXP under a different license than GPL which is then > bound to Python. Agreed. I think it would be really nice to have an interface for RXP that was easy to build and use. I haven't looked at PyLTXML in a long time, so I'm not sure what state it's in. > Given the many alternatives, I am not sure whether going with > expat is the right path... may be wrong though. As Martin said, RXP and Expat together don't really qualify as "many". sgmlop just isn't robust enough (yet), and it's not clear there are other alternatives. There is libxml (a.k.a. gnome-xml), which is licensed under the LGPL; Python bindings for that are described as being in the alpha stage, but I haven't had time to play with them myself. -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> PythonLabs at Zope Corporation From mal@lemburg.com Mon Oct 1 15:50:45 2001 From: mal@lemburg.com (M.-A. Lemburg) Date: Mon, 01 Oct 2001 16:50:45 +0200 Subject: [Python-Dev] Integrating Expat References: <200109301453.QAA21436@paros.informatik.hu-berlin.de> <3BB74B99.3B230398@lemburg.com> <200109301708.TAA22863@paros.informatik.hu-berlin.de> Message-ID: <3BB882C5.D1BDFE79@lemburg.com> Martin von Loewis wrote: > > > Are you sure that we should choose expat as "native" XML parser ? > > It wouldn't necessarily be the only parser. To process XML, different > applications have different needs. However, since the expatreader is > the only SAX reader included in the standard library at the moment, > guaranteeing presence of pyexpat is oft-requested. Notice that > pyexpat.c is also in the standard library already. Just wanted to make sure that we still have the option of including other parsers as well :-) > > There are other candidates which would fit this role just > > as well (in particular, Fredrik's sgmlop looks like a nice > > extension since it not only works with XML but also many > > other meta languages). > > Not that many candidates would work as well. For example, sgmlop has a > number of known bugs, and a few unknown ones. Guido once complained > that it is easy to crash sgmlop with ill-formed input, and rejected > inclusion of sgmlop when xmlrpclib was integrated. A known problem is > that entity references are not expanded in attributes. Well, let's put it this way: if someone finds a need to fix these bugs, it is more likely to happen in the Python core, e.g. xmlrpclib has already received a few tweaks (by yourself ;-) after it was checked into the core. I think that the sgmlop design is sufficiently simple and easy to extend to make it a good candidate for inclusion. Sure, we'll get bug reports, but why not add sgmlop marked as experimental to the core in order to get it stabilized and bug-fixed ?! I would very much like a sandbox like part in the Python standard dist to encourage stabilizing of proposed-to-be-included std lib extensions, e.g. how about a sandbox package in the std lib ?! > Beyond that, I'm not aware of many more pure-C parsers that could be > reasonably be integrated into the core. There are many XML parsers, > but many of the are written in C++ or Java. Me neither... except RXP which is written in plain C. > > If you want a very fast validating XML parser, RXP would also > > be a good choice -- AFAIK, the RXP folks would allow us to > > ship RXP under a different license than GPL which is then > > bound to Python. > > RXP would indeed be a choice. Of course, integrating it is much > harder; you'd have to write the C module first, plus documentation, > plus a SAX driver, plus test cases. I'm not sure how much code you can > inherit from PyLTXML. Sure; the question I wanted to raise was: given that we have such an interface, would RXP also be a candidate for inclusion ? > On performance: Please have a look at > > http://www.xml.com/lpt/a/Benchmark/exec.html > > which suggests that expat still has a speed advantage over rxp > (assuming that the measurements where done carefully, i.e. disabling > validation in RXP). Hmm, I know that at least one company has been having great success in using RXP with Python; from their experience RXP is faster on average XML than any of the other available (validating) parsers. May be due to their application, though, so YMMV. > > Given the many alternatives, I am not sure whether going with > > expat is the right path... may be wrong though. > > It shouldn't be the only path. pyexpat is already integrated into the > Python library, all I'm suggesting to give the promise that it will be > available on every 2.2 Python installation. > > Any volunteers working on RXP integration are certainly welcome to do > so; code contributions to PyXML will be welcome (provided the GPL > issue gets resolved). Code contributions to the Python core would > require some review, of course - it took quite some time to get > pyexpat stable, and I guess any other C-integrated parser won't work > from scratch, either. True. Thanks, -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Consulting & Company: http://www.egenix.com/ Python Software: http://www.lemburg.com/python/ From mal@lemburg.com Mon Oct 1 16:02:22 2001 From: mal@lemburg.com (M.-A. Lemburg) Date: Mon, 01 Oct 2001 17:02:22 +0200 Subject: [Python-Dev] Integrating Expat References: <200109301453.QAA21436@paros.informatik.hu-berlin.de> <3BB74B99.3B230398@lemburg.com> <200109301708.TAA22863@paros.informatik.hu-berlin.de> Message-ID: <3BB8857E.4C370439@lemburg.com> Martin von Loewis wrote: > > > If you want a very fast validating XML parser, RXP would also > > be a good choice -- AFAIK, the RXP folks would allow us to > > ship RXP under a different license than GPL which is then > > bound to Python. > > RXP would indeed be a choice. Of course, integrating it is much > harder; you'd have to write the C module first, plus documentation, > plus a SAX driver, plus test cases. I'm not sure how much code you can > inherit from PyLTXML. > > On performance: Please have a look at > > http://www.xml.com/lpt/a/Benchmark/exec.html > > which suggests that expat still has a speed advantage over rxp > (assuming that the measurements where done carefully, i.e. disabling > validation in RXP). How would libxml fit into this picture ? http://xmlsoft.org/ libxml is written in C as well and under the LGPL. There's also Apache's Xerces which is written in a portable subset of C++ (is probably to big though to be intergated into Python): http://xml.apache.org/xerces-c/ -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Consulting & Company: http://www.egenix.com/ Python Software: http://www.lemburg.com/python/ From fredrik@pythonware.com Mon Oct 1 16:23:56 2001 From: fredrik@pythonware.com (Fredrik Lundh) Date: Mon, 1 Oct 2001 17:23:56 +0200 Subject: [Python-Dev] Integrating Expat References: <200109301453.QAA21436@paros.informatik.hu-berlin.de><3BB74B99.3B230398@lemburg.com><200109301708.TAA22863@paros.informatik.hu-berlin.de> <15288.31458.735886.325225@grendel.zope.com> Message-ID: <019a01c14a8d$178d39a0$b3fa42d5@hagrid> > I have mixed feelings. There are really two things that we could > do: We could add Expat to our CVS repository, which means syncing a > bunch of files everytime a new Expat release comes out I thought MvL had already volunteered to do this? > > There are other candidates which would fit this role just > > as well (in particular, Fredrik's sgmlop looks like a nice > > extension since it not only works with XML but also many > > other meta languages). > > See Martin's comments about this. I think this precludes inclusion > of sgmlop until the problems it has have been addressed in the > implementation. cannot fix bugs if nobody bothers to report them ;-) (the crash issue appears to be a rumour; there was a bug when running in SGML mode, but that was fixed long ago. people using the current release in real-life applications haven't reported any stability problems...) on the other hand, sgmlop itself will never be anything but a "fast but sloppy" XML tokenizer. if you risk running into xml compliance nazis <0.1 wink>, you shouldn't use it. </F> From fdrake@acm.org Mon Oct 1 16:23:59 2001 From: fdrake@acm.org (Fred L. Drake, Jr.) Date: Mon, 1 Oct 2001 11:23:59 -0400 Subject: [Python-Dev] Integrating Expat In-Reply-To: <019a01c14a8d$178d39a0$b3fa42d5@hagrid> References: <200109301453.QAA21436@paros.informatik.hu-berlin.de> <3BB74B99.3B230398@lemburg.com> <200109301708.TAA22863@paros.informatik.hu-berlin.de> <15288.31458.735886.325225@grendel.zope.com> <019a01c14a8d$178d39a0$b3fa42d5@hagrid> Message-ID: <15288.35471.300115.993620@grendel.zope.com> Fredrik Lundh writes: > I thought MvL had already volunteered to do this? I didn't state this was a huge issue or that it didn't have a nice solution. ;-) It also isn't something that happens all that often, given that I don't have a lot of time to make Expat releases. > cannot fix bugs if nobody bothers to report them ;-) > > (the crash issue appears to be a rumour; there was a bug when > running in SGML mode, but that was fixed long ago. people using > the current release in real-life applications haven't reported any > stability problems...) Glad to hear this! Perhaps someone (not implying you) should start writing a substantial test suite for it to ferret out any remaining bugs? I don't see a test_sgmlop.py in the PyXML package; if you already have something perhaps you could contribute it? It might help you unload maintenance if anyone does manage to find a bug. > on the other hand, sgmlop itself will never be anything but a "fast > but sloppy" XML tokenizer. if you risk running into xml compliance > nazis <0.1 wink>, you shouldn't use it. "Nazi" would not have been my word for it, but ... Wham! ;-) -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> PythonLabs at Zope Corporation From jepler@inetnebr.com Mon Oct 1 15:00:29 2001 From: jepler@inetnebr.com (Jeff Epler) Date: Mon, 1 Oct 2001 09:00:29 -0500 Subject: [Python-Dev] Python 2.2a* getattr suggestion and question In-Reply-To: <200109301837.OAA32659@cj20424-a.reston1.va.home.com> References: <Pine.LNX.4.30.0109301748460.8078-100000@rnd.onego.ru> <200109301837.OAA32659@cj20424-a.reston1.va.home.com> Message-ID: <20011001090026.B1755@inetnebr.com> On Sun, Sep 30, 2001 at 02:37:00PM -0400, Guido van Rossum wrote: > > Sure. Still, I think interpreter diagnostics should be pointing to the > > exact place of trouble. At least, __getattribute__ must appear somewhere > > in the traceback to give a hint where from __repr__ was attempted to be > > called. > > When you write a faulty __getattribute__ that returns None instead of > raising AttributeError, it's not realistic to expect __getattribute__ > to be in the stack trace. PyChecker might want to give an error if flow in __getattr[ibute]__ doesn't either pass through a return or a 'raise AttributeError'. Even if the intent is to have attributes default to None, an explicit 'return None' could be added. Jeff From skip@pobox.com (Skip Montanaro) Mon Oct 1 18:08:00 2001 From: skip@pobox.com (Skip Montanaro) (Skip Montanaro) Date: Mon, 1 Oct 2001 12:08:00 -0500 Subject: [Python-Dev] Integrating Expat In-Reply-To: <3BB882C5.D1BDFE79@lemburg.com> References: <200109301453.QAA21436@paros.informatik.hu-berlin.de> <3BB74B99.3B230398@lemburg.com> <200109301708.TAA22863@paros.informatik.hu-berlin.de> <3BB882C5.D1BDFE79@lemburg.com> Message-ID: <15288.41712.751198.83416@beluga.mojam.com> mal> I think that the sgmlop design is sufficiently simple and easy to mal> extend to make it a good candidate for inclusion. Sure, we'll get mal> bug reports, but why not add sgmlop marked as experimental to the mal> core in order to get it stabilized and bug-fixed ?! I would be happy to sgmlop added to the core. The xmlrpclib encoding and decoding do need some sort of C-based acceleration to be usable: % python testxmlrpc.py testing with xmlrpclib 0.9.8 using FastParser 415 dumps per second 106 loads per second disabling fast parsers in xmlrpclib using SlowParser 412 dumps per second 16.1 loads per second FWIW, the xmlrpclib delivered with Python is substantially slower dumping data than the 0.9.x versions that have been around awhile, though its decoding performance degrades less without sgmlop. Compare the above with this: % PYTHONPATH=~/misc/python/python2 python testxmlrpc.py testing with xmlrpclib 1.0b3 using SgmlopParser 229 dumps per second 94.3 loads per second disabling fast parsers in xmlrpclib using ExpatParser 231 dumps per second 76.8 loads per second I haven't had or taken the time to investigate the difference yet. Skip From loewis@informatik.hu-berlin.de Mon Oct 1 18:09:25 2001 From: loewis@informatik.hu-berlin.de (Martin von Loewis) Date: Mon, 1 Oct 2001 19:09:25 +0200 (MEST) Subject: [Python-Dev] Integrating Expat In-Reply-To: <15288.31458.735886.325225@grendel.zope.com> (fdrake@acm.org) References: <200109301453.QAA21436@paros.informatik.hu-berlin.de> <3BB74B99.3B230398@lemburg.com> <200109301708.TAA22863@paros.informatik.hu-berlin.de> <15288.31458.735886.325225@grendel.zope.com> Message-ID: <200110011709.TAA13228@paros.informatik.hu-berlin.de> > I have mixed feelings. There are really two things that we could > do: We could add Expat to our CVS repository, which means syncing a > bunch of files everytime a new Expat release comes out, or we could > bundle the Expat sources with the Python source distribution when the > distribution is built, but not add them to CVS. This avoids the extra > files in CVS, but complicates construction of the distribution and > adds a new wrinkle to the configuration management. I'm in favour of putting it into the CVS. This will allow CVS users to build it, and it will allow proper integration and testing of required changes into setup.py (and Setup.dist; the current text has caused a lot of confusion in the past). Regards, Martin From mal@lemburg.com Mon Oct 1 18:50:31 2001 From: mal@lemburg.com (M.-A. Lemburg) Date: Mon, 01 Oct 2001 19:50:31 +0200 Subject: [Python-Dev] Integrating Expat References: <200109301453.QAA21436@paros.informatik.hu-berlin.de> <3BB74B99.3B230398@lemburg.com> <200109301708.TAA22863@paros.informatik.hu-berlin.de> <3BB882C5.D1BDFE79@lemburg.com> <15288.41712.751198.83416@beluga.mojam.com> Message-ID: <3BB8ACE7.B61A4881@lemburg.com> Skip Montanaro wrote: > > mal> I think that the sgmlop design is sufficiently simple and easy to > mal> extend to make it a good candidate for inclusion. Sure, we'll get > mal> bug reports, but why not add sgmlop marked as experimental to the > mal> core in order to get it stabilized and bug-fixed ?! > > I would be happy to sgmlop added to the core. The xmlrpclib encoding and > decoding do need some sort of C-based acceleration to be usable: > > % python testxmlrpc.py > testing with xmlrpclib 0.9.8 > using FastParser > 415 dumps per second > 106 loads per second > disabling fast parsers in xmlrpclib > using SlowParser > 412 dumps per second > 16.1 loads per second > > FWIW, the xmlrpclib delivered with Python is substantially slower dumping > data than the 0.9.x versions that have been around awhile, though its > decoding performance degrades less without sgmlop. Compare the above with > this: > > % PYTHONPATH=~/misc/python/python2 python testxmlrpc.py > testing with xmlrpclib 1.0b3 > using SgmlopParser > 229 dumps per second > 94.3 loads per second > disabling fast parsers in xmlrpclib > using ExpatParser > 231 dumps per second > 76.8 loads per second > > I haven't had or taken the time to investigate the difference yet. Hmm, you cannot really compare these numbers though, since the two runs use two different sets of parsers. Have you checked using SgmlopParser with the 0.9.8 version of xmlrpclib ? There's also a project on SF called py-xmlrpc which uses a C implementation as basis and is said to be much faster than xmlrpclib (at least that's what they quote on their web-page). -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Consulting & Company: http://www.egenix.com/ Python Software: http://www.lemburg.com/python/ From skip@pobox.com (Skip Montanaro) Mon Oct 1 19:05:35 2001 From: skip@pobox.com (Skip Montanaro) (Skip Montanaro) Date: Mon, 1 Oct 2001 13:05:35 -0500 Subject: [Python-Dev] Integrating Expat In-Reply-To: <3BB8ACE7.B61A4881@lemburg.com> References: <200109301453.QAA21436@paros.informatik.hu-berlin.de> <3BB74B99.3B230398@lemburg.com> <200109301708.TAA22863@paros.informatik.hu-berlin.de> <3BB882C5.D1BDFE79@lemburg.com> <15288.41712.751198.83416@beluga.mojam.com> <3BB8ACE7.B61A4881@lemburg.com> Message-ID: <15288.45167.465025.917304@beluga.mojam.com> mal> Hmm, you cannot really compare these numbers though, since the two mal> runs use two different sets of parsers. Have you checked using mal> SgmlopParser with the 0.9.8 version of xmlrpclib ? They are the same parser. I forgot to mention that. What is called "FastParser" in 0.9.8 is called SgmlopParser in the CVS version. That has a different thing called "FastParser". I believe it is the thing you can get by contacting Pythonware. mal> There's also a project on SF called py-xmlrpc which uses a C mal> implementation as basis and is said to be much faster than mal> xmlrpclib (at least that's what they quote on their web-page). Yes, it is. Amazingly enough, the guy who wrote it (Shilad Sen at Sourcelight Technologies) works in the same building I do (and it's a pretty small building). We had lunch last week and talked a bit about it. It doesn't yet do Unicode. I sent Shilad my little test script. He modified it to use his parser. His results suggest that py-xmlrpc is about as fast as cPickle. Skip From Samuele Pedroni <pedroni@inf.ethz.ch> Mon Oct 1 10:20:10 2001 From: Samuele Pedroni <pedroni@inf.ethz.ch> (Samuele Pedroni) Date: Mon, 1 Oct 2001 11:20:10 +0200 (MET DST) Subject: [Python-Dev] Re: descr PLAN.txt Message-ID: <200110010920.LAA07635@core.inf.ethz.ch> > > I think and it seems it contains some useful information > > for when we try to port the descr changes to jython, > > or will all this stuff be completely documented somewhere else? > > Hopefully the PEPs will be much more complete records. PLAN.txt was > just my own to-do list, indicating how far I was along the realization > of the PEPs. I realize that the PEPs are currently behind, but it is > in the PLAN.txt to remedy this. :-) I see. > I'm glad you are planning to copy this effort in Jython. Yes I'm planning to do try that. I don't think that without keeping up with Python Jython can survive. Not that we are very fast at keeping up OTOH. (yes for very practical reasons I'm among those who would prefer a slower development for Python :)) > (It should > actually be simpler in Jython as it doesn't have so much of a > dichotomy between types and classes as C-Python does.) I take this as an encouragement but actually the good point is that Java is bit more productive than multi-platform C, that's all. You should consider that we have to integrate the changes with the whole Java interoperability architecture and that we should deal with the limitations of java reflection and probably invent some new kind of inheriting from java, different from what is already there, that means new code that should produce dynamically some java bytecode ... > If there's > anything I can do to help (short of coding in Java :-) please let me > know. In particular, if there's anything unclear in the PEPs or > PLAN.txt or in python.org/2.2/descrintro.html or anywhere else > (including the C code), I'd be happy to clarify it at your (or anybody > else's) request. Thanks, I imagine we will have some questions :) > (Is Finn Bock still around in Jython?) AFAIK yes. For 2.1 (yet not out) he has done: - rich comparison - new coercion model - weak ref support - importing from jars on sys.path ... I have done: static nested scopes fixed some old nasty bugs improved the compatibility of importing wrt CPython And in any case I hope so: given that for 2.2 we have to do: iterators generators descr both in the runtime and jythonc ... and our user base so far has been quite passive. Sorry for the rants. regards. From skip@pobox.com (Skip Montanaro) Mon Oct 1 18:55:37 2001 From: skip@pobox.com (Skip Montanaro) (Skip Montanaro) Date: Mon, 1 Oct 2001 12:55:37 -0500 Subject: [Python-Dev] xmlrpclib speed boost Message-ID: <15288.44569.922160.604530@beluga.mojam.com> Regarding slowdown in xmlrpclib dump performance, I wrote: Skip> I haven't had or taken the time to investigate the difference yet. Well, it turned out to be pretty trivial. Somewhere along the way, the import of the escape function from the cgi module got moved out of the top level into the various functions that call it. Moving it back out to top level fixed that. I just checked in this change and a fairly trivial test case for xmlrpclib. This change may be deemed not to be the correct fix as far backwards compatibility is concerned (it uses the "from m import x as y" feature which was new with 2.0 I think). If someone alters this fix, please don't put the import back into the functions that call cgi.escape. Skip From thomas@xs4all.net Mon Oct 1 19:45:49 2001 From: thomas@xs4all.net (Thomas Wouters) Date: Mon, 1 Oct 2001 20:45:49 +0200 Subject: [Python-Dev] xmlrpclib speed boost In-Reply-To: <15288.44569.922160.604530@beluga.mojam.com> References: <15288.44569.922160.604530@beluga.mojam.com> Message-ID: <20011001204548.B846@xs4all.nl> On Mon, Oct 01, 2001 at 12:55:37PM -0500, Skip Montanaro wrote: > This change may be deemed not to be the correct fix as far backwards > compatibility is concerned (it uses the "from m import x as y" feature which > was new with 2.0 I think). If someone alters this fix, please don't put the > import back into the functions that call cgi.escape. Why not ? Moving the import to the top level just causes the slowdown to occur at a different moment. If this is really the problem, the slowdown should occur only the first time you use a particular function (unless you explicitly un-import cgi somehow ?) Importing cgi only in the functions that actually use it, in order to avoid the slowdown unless it's really necessary, sure seems like a sensible solution to me :-) -- Thomas Wouters <thomas@xs4all.net> Hi! I'm a .signature virus! copy me into your .signature file to help me spread! From mal@lemburg.com Mon Oct 1 19:44:37 2001 From: mal@lemburg.com (M.-A. Lemburg) Date: Mon, 01 Oct 2001 20:44:37 +0200 Subject: [Python-Dev] Integrating Expat References: <200109301453.QAA21436@paros.informatik.hu-berlin.de> <3BB74B99.3B230398@lemburg.com> <200109301708.TAA22863@paros.informatik.hu-berlin.de> <3BB882C5.D1BDFE79@lemburg.com> <15288.41712.751198.83416@beluga.mojam.com> <3BB8ACE7.B61A4881@lemburg.com> <15288.45167.465025.917304@beluga.mojam.com> Message-ID: <3BB8B995.3EA05D66@lemburg.com> Skip Montanaro wrote: > > mal> Hmm, you cannot really compare these numbers though, since the two > mal> runs use two different sets of parsers. Have you checked using > mal> SgmlopParser with the 0.9.8 version of xmlrpclib ? > > They are the same parser. I forgot to mention that. What is called > "FastParser" in 0.9.8 is called SgmlopParser in the CVS version. That has a > different thing called "FastParser". I believe it is the thing you can get > by contacting Pythonware. I see. Rereading your numbers suggests that only dumps got slower. Now that you've fixed this in CVS the reason is obvious... from xyz import abc is slow. OTOH, Fredrik mentions that he put in this change in order to decrease startup time for the lib. I guess you can't win 'em all :-) > mal> There's also a project on SF called py-xmlrpc which uses a C > mal> implementation as basis and is said to be much faster than > mal> xmlrpclib (at least that's what they quote on their web-page). > > Yes, it is. Amazingly enough, the guy who wrote it (Shilad Sen at > Sourcelight Technologies) works in the same building I do (and it's a pretty > small building). We had lunch last week and talked a bit about it. It > doesn't yet do Unicode. I sent Shilad my little test script. He modified > it to use his parser. His results suggest that py-xmlrpc is about as fast > as cPickle. Cool ! -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Consulting & Company: http://www.egenix.com/ Python Software: http://www.lemburg.com/python/ From tim.one@home.com Mon Oct 1 19:47:04 2001 From: tim.one@home.com (Tim Peters) Date: Mon, 1 Oct 2001 14:47:04 -0400 Subject: [Python-Dev] xmlrpclib speed boost In-Reply-To: <15288.44569.922160.604530@beluga.mojam.com> Message-ID: <LNBBLJKPBEHFEDALKOLCIELHLOAA.tim.one@home.com> [Skip Montanaro] > ... > Somewhere along the way, the import of the escape function from the cgi > module got moved out of the top level into the various functions that > call it. Moving it back out to top level fixed that. I just checked in > this change and a fairly trivial test case for xmlrpclib. > > This change may be deemed not to be the correct fix as far backwards > compatibility is concerned (it uses the "from m import x as y" > feature which was new with 2.0 I think). I pay no attention to "backwards compatibility" issues in the libraries we ship, and have Guido's blessing for that ornery attitude: the libs we ship should do the best possible job in the context of the release they're shipped in. It's explicitly not a PythonLabs goal that people be able to pick modules out of a release and use them in an earlier release (although it may be a goal of some developers for some modules -- but then the burden is on them to ensure it works for the module and release combinations they care about). From skip@pobox.com (Skip Montanaro) Mon Oct 1 19:56:58 2001 From: skip@pobox.com (Skip Montanaro) (Skip Montanaro) Date: Mon, 1 Oct 2001 13:56:58 -0500 Subject: [Python-Dev] xmlrpclib speed boost In-Reply-To: <20011001204548.B846@xs4all.nl> References: <15288.44569.922160.604530@beluga.mojam.com> <20011001204548.B846@xs4all.nl> Message-ID: <15288.48250.206337.702825@beluga.mojam.com> >> This change may be deemed not to be the correct fix as far backwards >> compatibility is concerned (it uses the "from m import x as y" >> feature which was new with 2.0 I think). If someone alters this fix, >> please don't put the import back into the functions that call >> cgi.escape. Thomas> Why not ? Moving the import to the top level just causes the Thomas> slowdown to occur at a different moment. If this is really the Thomas> problem, the slowdown should occur only the first time you use a Thomas> particular function (unless you explicitly un-import cgi somehow Thomas> ?) Trust me, this really *is* the problem. Instead of getting imported once, it gets imported once for every string and once for every key in every dictionary. I timed it before and after. Compare Marshaller.dump_string before def dump_string(self, value): from cgi import escape self.write("<value><string>%s</string></value>\n" % escape(value)) and after def dump_string(self, value): self.write("<value><string>%s</string></value>\n" % _escape(value)) I don't care how little work it is to import a module a second time, it is probably going to be on the same order of magnitude as that write call. In Marshaller.dump_struct the import was *inside* the for loop. I originally pulled it up out of the for loop but left it inside the method. That got me about a 25% boost in my simple test. I was ready to check in that one change and thought, "aw hell, might as well see what happens if I pull the import all the way out to the top level". Dump performance went all the way back up to where 0.9.8 is. Thomas> Importing cgi only in the functions that actually use it, in Thomas> order to avoid the slowdown unless it's really necessary, sure Thomas> seems like a sensible solution to me :-) Except dumping structs (dictionaries) and strings happens a lot. Even if all we ever did was dump ints, floats and lists, the total cost of my change would be one import. Skip From guido@python.org Mon Oct 1 19:54:34 2001 From: guido@python.org (Guido van Rossum) Date: Mon, 01 Oct 2001 14:54:34 -0400 Subject: [Python-Dev] xmlrpclib speed boost In-Reply-To: Your message of "Mon, 01 Oct 2001 20:45:49 +0200." <20011001204548.B846@xs4all.nl> References: <15288.44569.922160.604530@beluga.mojam.com> <20011001204548.B846@xs4all.nl> Message-ID: <200110011854.OAA07942@cj20424-a.reston1.va.home.com> > > This change may be deemed not to be the correct fix as far > > backwards compatibility is concerned (it uses the "from m import x > > as y" feature which was new with 2.0 I think). If someone alters > > this fix, please don't put the import back into the functions that > > call cgi.escape. > > Why not ? Moving the import to the top level just causes the slowdown to > occur at a different moment. If this is really the problem, the slowdown > should occur only the first time you use a particular function (unless you > explicitly un-import cgi somehow ?) > > Importing cgi only in the functions that actually use it, in order to avoid > the slowdown unless it's really necessary, sure seems like a sensible > solution to me :-) There's a cost to an import inside a function even if the module is already imported. If the function is very small and heavily used (as it apparently was in this case) the function can slow down considerably. It's kind of sad that the escape() function, which is only a few lines, comes from the cgi module which is thousands of lines that are not normally used except by actual CGI scripts. --Guido van Rossum (home page: http://www.python.org/~guido/) From fdrake@acm.org Mon Oct 1 20:20:37 2001 From: fdrake@acm.org (Fred L. Drake, Jr.) Date: Mon, 1 Oct 2001 15:20:37 -0400 Subject: [Python-Dev] xmlrpclib speed boost In-Reply-To: <200110011854.OAA07942@cj20424-a.reston1.va.home.com> References: <15288.44569.922160.604530@beluga.mojam.com> <20011001204548.B846@xs4all.nl> <200110011854.OAA07942@cj20424-a.reston1.va.home.com> Message-ID: <15288.49669.875132.906282@grendel.zope.com> Guido van Rossum writes: > It's kind of sad that the escape() function, which is only a few > lines, comes from the cgi module which is thousands of lines that are > not normally used except by actual CGI scripts. Yep. In this case, it probably makes more sense to call the escape() function from the xml.sax.saxutils module, though. The cgi version is for HTML and the xml.sax.saxutils version is for XML (big surprise). With only one argument, though, there's no difference. If we really want to deal with the startup time problem (regardless of which module we get the escape function from), we could do this: def _escape(s): global _escape import xml.sax.saxutils _escape = xml.sax.saxutils.escape return _escape(s) -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> PythonLabs at Zope Corporation From fredrik@pythonware.com Mon Oct 1 21:01:31 2001 From: fredrik@pythonware.com (Fredrik Lundh) Date: Mon, 1 Oct 2001 22:01:31 +0200 Subject: [Python-Dev] xmlrpclib speed boost References: <15288.44569.922160.604530@beluga.mojam.com> Message-ID: <02cc01c14ab3$e0124660$b3fa42d5@hagrid> skip wrote: > This change may be deemed not to be the correct fix as far backwards > compatibility is concerned (it uses the "from m import x as y" feature which > was new with 2.0 I think). If someone alters this fix, please don't put the > import back into the functions that call cgi.escape. you forgot to remove the comment at the top of the file that says it works with 1.5.2 and later. I've restored backwards compatibility, and tweaked the marshaller a little bit more. the new code is about 80% faster than 1.0b3 on my machine. ymmv, as usual. </F> From skip@pobox.com (Skip Montanaro) Mon Oct 1 21:04:11 2001 From: skip@pobox.com (Skip Montanaro) (Skip Montanaro) Date: Mon, 1 Oct 2001 15:04:11 -0500 Subject: [Python-Dev] xmlrpclib speed boost In-Reply-To: <02cc01c14ab3$e0124660$b3fa42d5@hagrid> References: <15288.44569.922160.604530@beluga.mojam.com> <02cc01c14ab3$e0124660$b3fa42d5@hagrid> Message-ID: <15288.52283.381456.704181@beluga.mojam.com> Fredrik> skip wrote: >> This change may be deemed not to be the correct fix as far backwards >> compatibility is concerned (it uses the "from m import x as y" >> feature which was new with 2.0 I think). If someone alters this fix, >> please don't put the import back into the functions that call >> cgi.escape. Fredrik> you forgot to remove the comment at the top of the file that Fredrik> says it works with 1.5.2 and later. Thanks, this is what I was referring to (of course, I didn't see the comment). Fredrik> I've restored backwards compatibility, and tweaked the Fredrik> marshaller a little bit more. the new code is about 80% faster Fredrik> than 1.0b3 on my machine. ymmv, as usual. Cool. Skip From gward@python.net Tue Oct 2 01:54:17 2001 From: gward@python.net (Greg Ward) Date: Mon, 1 Oct 2001 20:54:17 -0400 Subject: [Python-Dev] Demo/dns Message-ID: <20011001205417.A6563@cthulhu.mems-exchange.org> Demo/dns is pretty old, and has been superseded by Anthony Baxter's pydns project on SourceForge. At the very least, Demo/dns/README should point to http://sourceforge.net/projects/pydns instead of Anthony's old page for this project (dead URL). More radically, we could delete all the code in Demo/dns, and just leave behind a little README pointing at Anthony's project. I've *just* checked out his code, haven't dug in seriously yet, so I don't know if this is a good idea yet. Opinions? If I hear nothing, I'll just fix the URL in the README. Greg -- Greg Ward - Unix nerd gward@python.net http://starship.python.net/~gward/ A day for firm decisions!!!!! Or is it? From skip@pobox.com (Skip Montanaro) Tue Oct 2 02:45:10 2001 From: skip@pobox.com (Skip Montanaro) (Skip Montanaro) Date: Mon, 1 Oct 2001 20:45:10 -0500 Subject: [Python-Dev] Performance of various marshallers Message-ID: <15289.7206.266140.820041@beluga.mojam.com> I just tested each of the marshallers readily available to me. I dumped and loaded this object: ['MusicEntry', 'email': 'foo@bar.baz.spam', 'time': '7:30pm', 'tickets': '', 'program': '', 'state': 'MA', 'start': '2002-01-26', 'venueurl': '', 'country': '', 'performers': ['An Evening with Karen Savoca'], 'addressid': 7283, 'name': '', 'zip': '', 'city': 'Sudbury', 'info': 'Reservations required. Please call (978)443-3253 or e-mail Laurie at lalcorn@ultranet.com.', 'merchandise': [], 'event': '', 'keywords': ['.zyx.41'], 'submit_time': '2001-08-28', 'key': 325629, 'active': 1, 'end': '2002-01-26', 'address1': '', 'venue': 'Fox Run House Concerts', 'price': '$17', 'address3': '', 'address2': '', 'update_time': '2001-09-22:19:28:44'}] I don't claim this is typical data, but it is typical of the type of data I push through XML-RPC, so it's important to me. You can see why moving imports out of dump_string was so worthwhile. I would be happy to change the object being marshalled to better reflect what people think is "typical". All numbers in the following table are in encodings or decodings per second. All times were measured using time.clock. The number of times the encoding/decoding operation was performed was varied to give a reasonable total test time (approximately 5 seconds). Each test was run 3 times. The largest number is recorded below, rounded to three significant digits. encode decode ------ ------ marshal 25900 7830 cPickle 1230 149 xmlrpclib 0.9.8 w/ sgmlop 416 107 w/o sgmlop 415 16.3 xmlrpclib 1.0b4 w/ sgmlop 365 92.0 w/o sgmlop 363 74.9 py-xmlrpc 2780 2260 Skip From fdrake@acm.org Tue Oct 2 03:42:38 2001 From: fdrake@acm.org (Fred L. Drake, Jr.) Date: Mon, 1 Oct 2001 22:42:38 -0400 Subject: [Python-Dev] Performance of various marshallers In-Reply-To: <15289.7206.266140.820041@beluga.mojam.com> References: <15289.7206.266140.820041@beluga.mojam.com> Message-ID: <15289.10654.162126.780312@grendel.zope.com> Skip Montanaro writes: > total test time (approximately 5 seconds). Each test was run 3 times. The > largest number is recorded below, rounded to three significant digits. > > encode decode > ------ ------ > marshal 25900 7830 > cPickle 1230 149 Were the cPickle tests run in binary or original flavor? > xmlrpclib 0.9.8 > w/ sgmlop 416 107 > w/o sgmlop 415 16.3 <--+ > xmlrpclib 1.0b4 | > w/ sgmlop 365 92.0 | > w/o sgmlop 363 74.9 <--+ > py-xmlrpc 2780 2260 | | +---------------------------------------------------+ | +----> I presume that Expat was available for the second run and not for the first? These should probably be broken into three categories: sgmlop, expat, and xmllib. I also presume that py-xmlrpc never calls from C->Python during the parse phase, but I've not yet had a chance to look at this code. -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> PythonLabs at Zope Corporation From skip@pobox.com (Skip Montanaro) Tue Oct 2 03:52:58 2001 From: skip@pobox.com (Skip Montanaro) (Skip Montanaro) Date: Mon, 1 Oct 2001 21:52:58 -0500 Subject: [Python-Dev] is htmllib broken in 2.2a4? Message-ID: <15289.11274.567758.60276@beluga.mojam.com> Responding to a question in python-help about extracting links from web pages, I wrote a simple href printer: import htmllib, formatter class MyParser(htmllib.HTMLParser): def anchor_bgn(self, href, name, type): print href fmt = formatter.NullFormatter() parser = MyParser(fmt, verbose=1) parser.feed(open("tour01.html").read()) parser.close() When run using 2.2a4, it never prints anything. It outputs a list of hrefs when run with 2.1 or 1.6. Either there's a bug somewhere (in my code possibly, though it's pretty simple) or some semantics changed that I missed. I thought maybe the method resolution order change affected things, but htmllib.HTMLParser only uses single inheritance. When displaying help about htmllib.HTMLParser, pydoc.help does emit the method resolution order, which it doesn't generally seem to do: class HTMLParser(sgmllib.SGMLParser) | Method resolution order: | HTMLParser | sgmllib.SGMLParser | markupbase.ParserBase ... Skip From fdrake@acm.org Tue Oct 2 03:55:54 2001 From: fdrake@acm.org (Fred L. Drake, Jr.) Date: Mon, 1 Oct 2001 22:55:54 -0400 Subject: [Python-Dev] is htmllib broken in 2.2a4? In-Reply-To: <15289.11274.567758.60276@beluga.mojam.com> References: <15289.11274.567758.60276@beluga.mojam.com> Message-ID: <15289.11450.952344.374616@grendel.zope.com> Skip Montanaro writes: > When run using 2.2a4, it never prints anything. It outputs a list of hrefs > when run with 2.1 or 1.6. Either there's a bug somewhere (in my code > possibly, though it's pretty simple) or some semantics changed that I Sounds like a bug to me. Please file a bug report on SF including your code, and assign to me. Thanks! -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> PythonLabs at Zope Corporation From tim.one@home.com Tue Oct 2 04:05:11 2001 From: tim.one@home.com (Tim Peters) Date: Mon, 1 Oct 2001 23:05:11 -0400 Subject: [Python-Dev] is htmllib broken in 2.2a4? In-Reply-To: <15289.11274.567758.60276@beluga.mojam.com> Message-ID: <LNBBLJKPBEHFEDALKOLCKEMLLOAA.tim.one@home.com> [Skip Montanaro] > Responding to a question in python-help about extracting links from web > pages, I wrote a simple href printer: > > import htmllib, formatter > > class MyParser(htmllib.HTMLParser): > def anchor_bgn(self, href, name, type): > print href > > fmt = formatter.NullFormatter() > parser = MyParser(fmt, verbose=1) > parser.feed(open("tour01.html").read()) > parser.close() > > When run using 2.2a4, it never prints anything. It outputs a > list of hrefs when run with 2.1 or 1.6. Either there's a bug somewhere > (in my code possibly, though it's pretty simple) or some semantics > changed that I missed. Sorry, I don't know anything about that, and I don't know that code. Open a bug report! Sure doesn't sound right to me. > I thought maybe the method resolution order change affected things, The MRO hasn't changed for classic classes. Only for new classes (so if you don't derive from object, nothing about MRO changed). > but htmllib.HTMLParser only uses single inheritance. When displaying > help about htmllib.HTMLParser, pydoc.help does emit the method > resolution order, which it doesn't generally seem to do: I recently changed pydoc to display MRO if and only if there are more than two classes an attribute *could* come from (if there are no more than two classe involved, there's no possibility of confusion; but if there are more than two, confusion is possible). > class HTMLParser(sgmllib.SGMLParser) > | Method resolution order: > | HTMLParser > | sgmllib.SGMLParser > | markupbase.ParserBase > ... It listed MRO simply because more than 2 classes are possible attribute sources. From skip@pobox.com (Skip Montanaro) Tue Oct 2 04:06:04 2001 From: skip@pobox.com (Skip Montanaro) (Skip Montanaro) Date: Mon, 1 Oct 2001 22:06:04 -0500 Subject: [Python-Dev] Performance of various marshallers In-Reply-To: <15289.10654.162126.780312@grendel.zope.com> References: <15289.7206.266140.820041@beluga.mojam.com> <15289.10654.162126.780312@grendel.zope.com> Message-ID: <15289.12060.493061.267430@beluga.mojam.com> > total test time (approximately 5 seconds). Each test was run 3 times. The > largest number is recorded below, rounded to three significant digits. > > encode decode > ------ ------ > marshal 25900 7830 > cPickle 1230 149 Were the cPickle tests run in binary or original flavor? I wasn't aware of a "binary flavor". It's not mentioned in the online docs. I just called cPickle.dumps or cPickle.loads as appropriate. It looks like I should call them with a second binary flag. > xmlrpclib 0.9.8 > w/ sgmlop 416 107 > w/o sgmlop 415 16.3 <--+ > xmlrpclib 1.0b4 | > w/ sgmlop 365 92.0 | > w/o sgmlop 363 74.9 <--+ > py-xmlrpc 2780 2260 | | +---------------------------------------------------+ | +----> I presume that Expat was available for the second run and not for the first? These should probably be broken into three categories: sgmlop, expat, and xmllib. In 0.9.8 there are two parsers, fast (with sgmlop) and slow (without). I believe the ExpatParser was used in the second version. It doesn't really matter to me because they are all perform so abysmally. I also presume that py-xmlrpc never calls from C->Python during the parse phase, but I've not yet had a chance to look at this code. I don't know. I've not looked at the code, only the output. I have cc'd Shilad Sen on this thread. He should be able to tell us how py-xmlrpc gets such good performance. Skip From skip@pobox.com (Skip Montanaro) Tue Oct 2 04:21:16 2001 From: skip@pobox.com (Skip Montanaro) (Skip Montanaro) Date: Mon, 1 Oct 2001 22:21:16 -0500 Subject: [Python-Dev] Performance of various marshallers In-Reply-To: <15289.10654.162126.780312@grendel.zope.com> References: <15289.7206.266140.820041@beluga.mojam.com> <15289.10654.162126.780312@grendel.zope.com> Message-ID: <15289.12972.375469.537213@beluga.mojam.com> > encode decode > ------ ------ > cPickle 1230 149 Were the cPickle tests run in binary or original flavor? Setting the binary flag when dumping using cPickle boosts the encode rate to 4450 and the decode rate to 3940. Skip From andy@reportlab.com Tue Oct 2 06:53:40 2001 From: andy@reportlab.com (Andy Robinson) Date: Tue, 2 Oct 2001 06:53:40 +0100 Subject: [Python-Dev] RE: Python-Dev digest, Vol 1 #1637 - 11 msgs In-Reply-To: <E15o5WL-0005Y6-00@mail.python.org> Message-ID: <LKENLBBMDHMKBECHIAIAEEGJCAAA.andy@reportlab.com> > If you want a very fast validating XML parser, RXP would also > be a good choice -- AFAIK, the RXP folks would allow us to > ship RXP under a different license than GPL which is then > bound to Python. > > Given the many alternatives, I am not sure whether going with > expat is the right path... may be wrong though. > Lucky I tuned in. Reportlab has had great success with RXP. We have a python wrapper, pyRXP, with binaries available for several platforms. It is GPLed at present. They wish to keep GPL just in case someone big comes along and wants their code for ten million set-top boxes or something. However, I persuaded them to grant a license to let it be used through the Python binding under Python-like terms, as long as we invent the words and save them having to waste time on lawyers. They would even be happy for it to go into the Python distribution. And we're happy to maintain the wrapper and binaries for several platforms, which we have to do for our customers anyway. If one of the core Python team, who I know have long and painful experience of this stuff, would like to drop me a line, we can probably sort this out in a night. The other thing we found very useful was our representation. We make reports, and ML is a common data source; so our goal is typically to slurp XML into memory as fast as possible, with validation. We eventually hit on a 'tuple tree': each tag is represented as (tagname, attrs, list-of-children, spare) We get there about 6x faster than the fastest alternative parser we know, because all the work is done in C; with typical use of other parsers you call back into Python on every tag. The tree structure is a fraction of the size in memory of what gets created by models using objects for every node. It would be very easy to add this as an alternative interface to expat as well. So then Python users could have a choice of tree or events, and validating or non-validating, all done in C and in the standard distribution. Andy Robinson CEO/Chief Architect, Reportlab Inc. From effbot@telia.com Tue Oct 2 08:43:04 2001 From: effbot@telia.com (Fredrik Lundh) Date: Tue, 2 Oct 2001 09:43:04 +0200 Subject: [Python-Dev] Performance of various marshallers References: <15289.7206.266140.820041@beluga.mojam.com> <15289.10654.162126.780312@grendel.zope.com> Message-ID: <03aa01c14b15$e329ac00$b3fa42d5@hagrid> fred wrote: > > xmlrpclib 0.9.8 > > w/ sgmlop 416 107 > > w/o sgmlop 415 16.3 <--+ > > xmlrpclib 1.0b4 | > > w/ sgmlop 365 92.0 | > > w/o sgmlop 363 74.9 <--+ > > py-xmlrpc 2780 2260 | > | > +---------------------------------------------------+ > | > +----> I presume that Expat was available for the second run and not > for the first? These should probably be broken into three > categories: sgmlop, expat, and xmllib. footnote: 0.9.8 didn't support pyexpat. > I also presume that py-xmlrpc never calls from C->Python > during the parse phase, but I've not yet had a chance to look > at this code. does py-xmlrpc use a real XML parser? </F> From skip@pobox.com (Skip Montanaro) Tue Oct 2 13:29:45 2001 From: skip@pobox.com (Skip Montanaro) (Skip Montanaro) Date: Tue, 2 Oct 2001 07:29:45 -0500 Subject: [Python-Dev] Performance of various marshallers In-Reply-To: <03aa01c14b15$e329ac00$b3fa42d5@hagrid> References: <15289.7206.266140.820041@beluga.mojam.com> <15289.10654.162126.780312@grendel.zope.com> <03aa01c14b15$e329ac00$b3fa42d5@hagrid> Message-ID: <15289.45881.920165.427067@beluga.mojam.com> >> I also presume that py-xmlrpc never calls from C->Python during the >> parse phase, but I've not yet had a chance to look at this code. Fredrik> does py-xmlrpc use a real XML parser? I suspect not. It's special purpose is to parse or generate XML-RPC, so you know ahead of time that the end result is the only thing you need. Skip From akuchlin@mems-exchange.org Tue Oct 2 18:57:24 2001 From: akuchlin@mems-exchange.org (Andrew Kuchling) Date: Tue, 02 Oct 2001 13:57:24 -0400 Subject: [Python-Dev] Bookstore proceeds Message-ID: <E15oTnI-0005pO-00@ute.cnri.reston.va.us> I've received another cheque from Amazon with the last quarter's proceeds for the Python bookstore. The amount is just a bit less than $500; I'll contribute the few dollars to bring it up to $500 flat. Question: what do with the money? Right now the best candidate is to pay the Python10 conference fees for some of Jeffrey Elkner's students. I can't think of anything else to do with the money; anyone have any brilliant suggestions? --amk From guido@python.org Tue Oct 2 19:28:33 2001 From: guido@python.org (Guido van Rossum) Date: Tue, 02 Oct 2001 14:28:33 -0400 Subject: [Python-Dev] Bookstore proceeds In-Reply-To: Your message of "Tue, 02 Oct 2001 13:57:24 EDT." <E15oTnI-0005pO-00@ute.cnri.reston.va.us> References: <E15oTnI-0005pO-00@ute.cnri.reston.va.us> Message-ID: <200110021828.f92ISXk25410@odiug.digicool.com> > I've received another cheque from Amazon with the last quarter's > proceeds for the Python bookstore. The amount is just a bit less than > $500; I'll contribute the few dollars to bring it up to $500 flat. > > Question: what do with the money? Right now the best candidate is to > pay the Python10 conference fees for some of Jeffrey Elkner's > students. I can't think of anything else to do with the money; anyone > have any brilliant suggestions? Subsidize Finn Bock and Samuele Pedroni coming to Python10? --Guido van Rossum (home page: http://www.python.org/~guido/) From paul@ActiveState.com Tue Oct 2 19:34:45 2001 From: paul@ActiveState.com (Paul Prescod) Date: Tue, 02 Oct 2001 11:34:45 -0700 Subject: [Python-Dev] Performance of various marshallers References: <15289.7206.266140.820041@beluga.mojam.com> <15289.10654.162126.780312@grendel.zope.com> <03aa01c14b15$e329ac00$b3fa42d5@hagrid> <15289.45881.920165.427067@beluga.mojam.com> Message-ID: <3BBA08C5.240EB4BC@ActiveState.com> Skip Montanaro wrote: > >... > > I suspect not. It's special purpose is to parse or generate XML-RPC, so you > know ahead of time that the end result is the only thing you need. One reason to use a full XML parser is you get Unicode cheaply. I don't see Unicode as a feature that you add in a weekend at the end... Paul Prescod From akuchlin@mems-exchange.org Tue Oct 2 19:37:34 2001 From: akuchlin@mems-exchange.org (Andrew Kuchling) Date: Tue, 2 Oct 2001 14:37:34 -0400 Subject: [Python-Dev] Bookstore proceeds In-Reply-To: <200110021828.f92ISXk25410@odiug.digicool.com>; from guido@python.org on Tue, Oct 02, 2001 at 02:28:33PM -0400 References: <E15oTnI-0005pO-00@ute.cnri.reston.va.us> <200110021828.f92ISXk25410@odiug.digicool.com> Message-ID: <20011002143734.J21687@ute.mems-exchange.org> On Tue, Oct 02, 2001 at 02:28:33PM -0400, Guido van Rossum wrote: >Subsidize Finn Bock and Samuele Pedroni coming to Python10? Will $500 be of significant assistance? Note on cost: student fees seem to be around $250, so that's two students. Developer's Day registration is $295, so it could almost pay for their admission for that day; it's not enough to help with their hotel or flight. --amk From guido@python.org Tue Oct 2 19:41:09 2001 From: guido@python.org (Guido van Rossum) Date: Tue, 02 Oct 2001 14:41:09 -0400 Subject: [Python-Dev] Bookstore proceeds In-Reply-To: Your message of "Tue, 02 Oct 2001 14:37:34 EDT." <20011002143734.J21687@ute.mems-exchange.org> References: <E15oTnI-0005pO-00@ute.cnri.reston.va.us> <200110021828.f92ISXk25410@odiug.digicool.com> <20011002143734.J21687@ute.mems-exchange.org> Message-ID: <200110021841.f92If9T26208@odiug.digicool.com> > Will $500 be of significant assistance? Ask them. > Note on cost: student fees seem to be around $250, so that's two > students. Developer's Day registration is $295, so it could almost > pay for their admission for that day; it's not enough to help with > their hotel or flight. Flights are cheap. Somone locally might be able to put them up. --Guido van Rossum (home page: http://www.python.org/~guido/) From skip@pobox.com (Skip Montanaro) Tue Oct 2 19:57:25 2001 From: skip@pobox.com (Skip Montanaro) (Skip Montanaro) Date: Tue, 2 Oct 2001 13:57:25 -0500 Subject: [Python-Dev] Performance of various marshallers In-Reply-To: <3BBA08C5.240EB4BC@ActiveState.com> References: <15289.7206.266140.820041@beluga.mojam.com> <15289.10654.162126.780312@grendel.zope.com> <03aa01c14b15$e329ac00$b3fa42d5@hagrid> <15289.45881.920165.427067@beluga.mojam.com> <3BBA08C5.240EB4BC@ActiveState.com> Message-ID: <15290.3605.641031.46951@beluga.mojam.com> >> I suspect not. It's special purpose is to parse or generate XML-RPC, >> so you know ahead of time that the end result is the only thing you >> need. Paul> One reason to use a full XML parser is you get Unicode cheaply. I Paul> don't see Unicode as a feature that you add in a weekend at the Paul> end... XML-RPC's relationship to Unicode is ill-defined. The spec that Dave Winer wrote requires all data to be US-ASCII, so XML-RPC isn't really XML-compliant. (You'll have to take up issues of standards compliance with Dave.) Still, Unicode or not, the notion that XML-RPC is a data serialization mechanism instead of a compound data markup language means you don't need to provide hooks for processing each element, so full-blown XML parsers tend to be overkill as py-xmlrpc demonstrates. No matter how hard Shilad finds it to add Unicode support to his package, it's still likely to be miles ahead of other XML parsers. Skip From bckfnn@worldonline.dk Tue Oct 2 20:12:53 2001 From: bckfnn@worldonline.dk (Finn Bock) Date: Tue, 02 Oct 2001 19:12:53 GMT Subject: [Python-Dev] Bookstore proceeds In-Reply-To: <200110021828.f92ISXk25410@odiug.digicool.com> References: <E15oTnI-0005pO-00@ute.cnri.reston.va.us> <200110021828.f92ISXk25410@odiug.digicool.com> Message-ID: <3bba10cd.14259694@mail.wanadoo.dk> [Andrew Kuchling] > Question: what do with the money? Right now the best candidate is to > pay the Python10 conference fees for some of Jeffrey Elkner's > students. I can't think of anything else to do with the money; anyone > have any brilliant suggestions? [Guido] >Subsidize Finn Bock and Samuele Pedroni coming to Python10? "Thanks, but No Thanks" from me. While I would dearly like to meet you all, the travel is too much of a pain for my taste. regards, finn From martin@loewis.home.cs.tu-berlin.de Tue Oct 2 21:09:51 2001 From: martin@loewis.home.cs.tu-berlin.de (Martin v. Loewis) Date: Tue, 2 Oct 2001 22:09:51 +0200 Subject: [Python-Dev] Demo/dns Message-ID: <200110022009.f92K9p301782@mira.informatik.hu-berlin.de> > More radically, we could delete all the code in Demo/dns, and just > leave behind a little README pointing at Anthony's project. +1. Martin From guido@python.org Tue Oct 2 21:18:59 2001 From: guido@python.org (Guido van Rossum) Date: Tue, 02 Oct 2001 16:18:59 -0400 Subject: [Python-Dev] Demo/dns In-Reply-To: Your message of "Tue, 02 Oct 2001 22:09:51 +0200." <200110022009.f92K9p301782@mira.informatik.hu-berlin.de> References: <200110022009.f92K9p301782@mira.informatik.hu-berlin.de> Message-ID: <200110022018.f92KIxV29996@odiug.digicool.com> > > More radically, we could delete all the code in Demo/dns, and just > > leave behind a little README pointing at Anthony's project. [MvL] > +1. If nobody disagrees, I'll do this. It's unclear where to leave a pointer -- a Demo/dns directory without code feels a little strange. Maybe we don't need to leave a pointer? --Guido van Rossum (home page: http://www.python.org/~guido/) From paul@ActiveState.com Tue Oct 2 21:44:11 2001 From: paul@ActiveState.com (Paul Prescod) Date: Tue, 02 Oct 2001 13:44:11 -0700 Subject: [Python-Dev] Performance of various marshallers References: <15289.7206.266140.820041@beluga.mojam.com> <15289.10654.162126.780312@grendel.zope.com> <03aa01c14b15$e329ac00$b3fa42d5@hagrid> <15289.45881.920165.427067@beluga.mojam.com> <3BBA08C5.240EB4BC@ActiveState.com> <15290.3605.641031.46951@beluga.mojam.com> Message-ID: <3BBA271B.63B07CEF@ActiveState.com> Skip Montanaro wrote: > >... > > XML-RPC's relationship to Unicode is ill-defined. The spec that Dave Winer > wrote requires all data to be US-ASCII, so XML-RPC isn't really > XML-compliant. (You'll have to take up issues of standards compliance with > Dave.) Most XML-RPC implementations support Unicode, Dave Winer notwithstanding. Plus, the XML-RPC spec says nothing to indicate that XML-RPC documents may not be encoded in either of XML's two built-in encodings (even if the data is restricted to ASCII values). > Still, Unicode or not, the notion that XML-RPC is a data serialization > mechanism instead of a compound data markup language means you don't need to > provide hooks for processing each element, so full-blown XML parsers tend to > be overkill as py-xmlrpc demonstrates. I don't see how that follows. py-xmlrpc needs to handle <struct> different than <array> so it needs to have a "hook" for each of those element types. Having a fixed list of hooks or an extensible array of them should not be much different from a performance point of view. Yes, an incomplete XML parser could be faster if it ignores Unicode, ignores character references, and does not do all of the error checking required by the spec. I'm not sure if this would really improve performance anyhow. py-xmlrpc is probably faster because it doesn't call out to Python code until the entire message has been parsed. xmlrpclib on the other hand, is entirely written in Python. Is there a Python XML-RPC implementation that uses no Python code but does use a true XML parser? > ... No matter how hard Shilad finds it > to add Unicode support to his package, it's still likely to be miles ahead > of other XML parsers. I think you are exaggerating the benefit of having a fixed vocabulary. There is hardly any performance boost possible based on that one detail. Paul Prescod From martin@loewis.home.cs.tu-berlin.de Tue Oct 2 22:30:13 2001 From: martin@loewis.home.cs.tu-berlin.de (Martin v. Loewis) Date: Tue, 2 Oct 2001 23:30:13 +0200 Subject: [Python-Dev] Integrating Expat Message-ID: <200110022130.f92LUDt02414@mira.informatik.hu-berlin.de> > How would libxml fit into this picture ? Uncertain. I could find out the following facts: Python Wrappers: Dave Kuhlmann has written some, see http://www.rexx.com/~dkuhlman/ Licensing: Available through either LGPL or W3C IPR (Daniel Veillard from Redhat is the maintainer, he used to work for the W3C). Portability: Apparently tested on Unix and Win32. Uses zlib and iconv when available. Size: 85kLOC C source code (compare to 11kLOC in Expat) Performance: I could not find any results on this subject, comparing libxml with other parsers. Regards, Martin From nas@python.ca Tue Oct 2 22:51:14 2001 From: nas@python.ca (Neil Schemenauer) Date: Tue, 2 Oct 2001 14:51:14 -0700 Subject: [Python-Dev] Demo/dns In-Reply-To: <200110022018.f92KIxV29996@odiug.digicool.com>; from guido@python.org on Tue, Oct 02, 2001 at 04:18:59PM -0400 References: <200110022009.f92K9p301782@mira.informatik.hu-berlin.de> <200110022018.f92KIxV29996@odiug.digicool.com> Message-ID: <20011002145114.A20139@glacier.arctrix.com> Guido van Rossum wrote: > It's unclear where to leave a pointer -- a Demo/dns directory without > code feels a little strange. Misc/NEWS? Neil From skip@pobox.com (Skip Montanaro) Tue Oct 2 22:53:15 2001 From: skip@pobox.com (Skip Montanaro) (Skip Montanaro) Date: Tue, 2 Oct 2001 16:53:15 -0500 Subject: [Python-Dev] Performance of various marshallers In-Reply-To: <3BBA271B.63B07CEF@ActiveState.com> References: <15289.7206.266140.820041@beluga.mojam.com> <15289.10654.162126.780312@grendel.zope.com> <03aa01c14b15$e329ac00$b3fa42d5@hagrid> <15289.45881.920165.427067@beluga.mojam.com> <3BBA08C5.240EB4BC@ActiveState.com> <15290.3605.641031.46951@beluga.mojam.com> <3BBA271B.63B07CEF@ActiveState.com> Message-ID: <15290.14155.432881.855383@beluga.mojam.com> >> Still, Unicode or not, the notion that XML-RPC is a data >> serialization mechanism instead of a compound data markup language >> means you don't need to provide hooks for processing each element, so >> full-blown XML parsers tend to be overkill as py-xmlrpc demonstrates. Paul> I don't see how that follows. py-xmlrpc needs to handle <struct> Paul> different than <array> so it needs to have a "hook" for each of Paul> those element types. Having a fixed list of hooks or an extensible Paul> array of them should not be much different from a performance Paul> point of view. Sure, <struct> and <array> mean different things, but <struct> will always mean the same thing in an XML-RPC context. There's no need to provide any hooks. Once you've successfully parsed a <struct> you get a Python dictionary. As far as I can tell sgmlop is always going to be slower than py-xmlrpc because it must callback to an Unmarshaller instance for each tag. The only option currently available is the Unmarshaller class written in Python. Pythonware has a FastParser/FastUnmarshaller pair available now which I don't have access to. Perhaps it exhibits encode/decode speeds similar to py-xmlrpc. You'll have to ask Fredrik. Py-xmlrpc was written with the knowledge that intermediate results aren't useful and that as you put it, it has a fixed vocabulary. Why structure a parser to accommodate situations that aren't needed? Paul> Yes, an incomplete XML parser could be faster if it ignores Paul> Unicode, ignores character references, and does not do all of the Paul> error checking required by the spec. I'm not sure if this would Paul> really improve performance anyhow. Does py-xmlrpc have a ways to go? Sure. It's still pretty new software, so give it time. You seem to be dismissing it completely because it's not as mature as, say, Expat. I doubt it will lose a factor of 8 in encoding speed or a factor of 24 in decoding speed (the current speed advantages I measure over xmlrpclib 1.0b4 using sgmlop) when those things are all added. I'm not sure all those things will ever be needed, but you're welcome to think they will. Paul> py-xmlrpc is probably faster because it doesn't call out to Python Paul> code until the entire message has been parsed. xmlrpclib on the Paul> other hand, is entirely written in Python. Is there a Python Paul> XML-RPC implementation that uses no Python code but does use a Paul> true XML parser? That's precisely why py-xmlrpc is faster. Should it behave some other way? I don't think there is another XML-RPC parser out there that is available from Python but that doesn't use Python. >> ... No matter how hard Shilad finds it to add Unicode support to his >> package, it's still likely to be miles ahead of other XML parsers. Paul> I think you are exaggerating the benefit of having a fixed Paul> vocabulary. There is hardly any performance boost possible based Paul> on that one detail. I don't understand see how you can't make that connection. XML-RPC has a fixed vocabulary and never needs to look at intermediate results. It sounds to me like all you have is a hammer so everything looks like a nail. There are places for general-purpose XML parsers and places for special-purpose XML parsers. In this particular context I only care about how fast I can push objects between a client and server using XML-RPC. I apologize if the subject seems more general than I intended. My only intention was to compare the data serialization performance of various tools. I didn't include "XML-RPC" in the subject of this thread because I tossed in marshal and cPickle results as well, simply for comparison. Skip From pedroni@inf.ethz.ch Tue Oct 2 22:57:35 2001 From: pedroni@inf.ethz.ch (Samuele Pedroni) Date: Tue, 2 Oct 2001 23:57:35 +0200 Subject: [Python-Dev] Bookstore proceeds References: <E15oTnI-0005pO-00@ute.cnri.reston.va.us> <200110021828.f92ISXk25410@odiug.digicool.com> Message-ID: <00d601c14b8d$40049a80$8a73fea9@newmexico> [Guido] > Subsidize Finn Bock and Samuele Pedroni coming to Python10? Thanks for the proposal. Surely I would like to meet you all but - financial issues aside - while nice for me, I don't think my presence would be that much constructive and Finn - who's the main mantainer and has done a whole lot - deserves more than me to eventually meet you and the users and receive their thanks. I'm not your guy and Finn don't want to fly :( Samuele. From fredrik@pythonware.com Tue Oct 2 23:18:02 2001 From: fredrik@pythonware.com (Fredrik Lundh) Date: Wed, 3 Oct 2001 00:18:02 +0200 Subject: [Python-Dev] Performance of various marshallers References: <15289.7206.266140.820041@beluga.mojam.com> <15289.10654.162126.780312@grendel.zope.com> <03aa01c14b15$e329ac00$b3fa42d5@hagrid> <15289.45881.920165.427067@beluga.mojam.com> <3BBA08C5.240EB4BC@ActiveState.com> <15290.3605.641031.46951@beluga.mojam.com> <3BBA271B.63B07CEF@ActiveState.com> Message-ID: <050d01c14b90$213834b0$b3fa42d5@hagrid> paul wrote: > > XML-RPC's relationship to Unicode is ill-defined. The spec that Dave Winer > > wrote requires all data to be US-ASCII, so XML-RPC isn't really > > XML-compliant. (You'll have to take up issues of standards compliance with > > Dave.) > > Most XML-RPC implementations support Unicode, Dave Winer > notwithstanding. Plus, the XML-RPC spec says nothing to indicate that > XML-RPC documents may not be encoded in either of XML's two built-in > encodings (even if the data is restricted to ASCII values). the specification says that XML-RPC uses XML and HTTP. it doesn't say anything about a Dave-specific subset of XML or HTTP... (like so many other parts of the specification, the "string" type isn't exactly well-specified. the specification first says that strings contains ASCII characters, and later that "any characters are allowed in a string" and that "a string can be used to encode binary data") > Yes, an incomplete XML parser could be faster if it ignores Unicode, > ignores character references, and does not do all of the error checking > required by the spec. I'm not sure if this would really improve > performance anyhow. well, sgmlop is a bit faster than expat (up to 50%, in some tests). expat does a bit more error checking. > xmlrpclib on the other hand, is entirely written in Python. Is there a > Python XML-RPC implementation that uses no Python code but does > use a true XML parser? the _xmlrpclib accelerator (see the xmlrpclib.py source) uses expat, with a really fast C layer. judging from Skip's benchmarks, expat is a bit slower than the py-xmlrpc parser (which is why I asked). </F> From paul@pfdubois.com Tue Oct 2 23:16:16 2001 From: paul@pfdubois.com (Paul F. Dubois) Date: Tue, 2 Oct 2001 15:16:16 -0700 Subject: [Python-Dev] Pmw broken under 2.2a4? Message-ID: <000101c14b8f$db4938a0$3d01a8c0@plstn1.sfba.home.com> Building Pmw 0.8.5 under 2.2a4 I got syntax errors of the 'inconsistent use of tabs and spaces' kind. I'm not sure who I should tell but I speculate that I just did. From paul@ActiveState.com Tue Oct 2 23:15:49 2001 From: paul@ActiveState.com (Paul Prescod) Date: Tue, 02 Oct 2001 15:15:49 -0700 Subject: [Python-Dev] Performance of various marshallers References: <15289.7206.266140.820041@beluga.mojam.com> <15289.10654.162126.780312@grendel.zope.com> <03aa01c14b15$e329ac00$b3fa42d5@hagrid> <15289.45881.920165.427067@beluga.mojam.com> <3BBA08C5.240EB4BC@ActiveState.com> <15290.3605.641031.46951@beluga.mojam.com> <3BBA271B.63B07CEF@ActiveState.com> <15290.14155.432881.855383@beluga.mojam.com> Message-ID: <3BBA3C95.1CC094CC@ActiveState.com> Skip Montanaro wrote: > >... > > Sure, <struct> and <array> mean different things, but <struct> will always > mean the same thing in an XML-RPC context. There's no need to provide any > hooks. There are two different issues. One is parsing: taking a string of bytes and interpreting them as XML. The other is passing this information to the Python programmer. The handling of "hooks" is on the backend, passing the information to the Python programmer. I interpreted Fredrick's question as being about the front end: does it use a real XML parser or not. >... > Does py-xmlrpc have a ways to go? Sure. It's still pretty new software, so > give it time. You seem to be dismissing it completely because it's not as > mature as, say, Expat. I'm not asking it to be as mature as Expat. I'm asking why it didn't *use* Expat or some other parser. Expat would recognize structs and arrays and pass them to C code which builds Python objects. Then those Python objects can be passed to Python. >... > Paul> py-xmlrpc is probably faster because it doesn't call out to Python > Paul> code until the entire message has been parsed. xmlrpclib on the > Paul> other hand, is entirely written in Python. Is there a Python > Paul> XML-RPC implementation that uses no Python code but does use a > Paul> true XML parser? > > That's precisely why py-xmlrpc is faster. Should it behave some other way? > I don't think there is another XML-RPC parser out there that is available > from Python but that doesn't use Python. Okay, so we agree that the fast part is probably not so much the parser but the handing of data to Python. So why rewrite a parser? Nothing requires an Expat-using XML-RPC implementation to call back into Python for every element. It can collect the results in C and then call Python when it has values. >... > I don't understand see how you can't make that connection. XML-RPC has a > fixed vocabulary and never needs to look at intermediate results. Let me suggest an analogy. Someone writes "CGIPython". It uses a specially optimized parser designed for parsing only Python CGI scripts. Do you think it would run much faster than the regular Python parser? Well, syntactically CGI scripts are basically the same as ordinary Python programs so why would you *want* a specialized parser? Parsing angle brackets is the same whether they are in an XML-RPC message or a Docbook document, just as parsing Python is the same, whether it is a CGI or a GUI app. > ... It sounds > to me like all you have is a hammer so everything looks like a nail. There > are places for general-purpose XML parsers and places for special-purpose > XML parsers. In this particular context I only care about how fast I can > push objects between a client and server using XML-RPC. I don't personally see much benefit using XML if you don't adhere to the XML spec. Just perusing the code quickly I believe I've found a few bugs that it would not have had if it built on Expat or some other XML parser. 1. It doesn't handle ? syntax. 2. It doesn't handle <methodCall > (extra whitespace) 3. I strongly suspect it won't handle comments in the XML. 4. It won't handle the mandatory UTF-16 encoding from XML 5. It won't handle CDATA sections. Paul Prescod From paul@ActiveState.com Tue Oct 2 23:19:05 2001 From: paul@ActiveState.com (Paul Prescod) Date: Tue, 02 Oct 2001 15:19:05 -0700 Subject: [Python-Dev] Performance of various marshallers References: <15289.7206.266140.820041@beluga.mojam.com> <15289.10654.162126.780312@grendel.zope.com> <03aa01c14b15$e329ac00$b3fa42d5@hagrid> <15289.45881.920165.427067@beluga.mojam.com> <3BBA08C5.240EB4BC@ActiveState.com> <15290.3605.641031.46951@beluga.mojam.com> <3BBA271B.63B07CEF@ActiveState.com> <050d01c14b90$213834b0$b3fa42d5@hagrid> Message-ID: <3BBA3D59.441625E9@ActiveState.com> Fredrik Lundh wrote: > >... > > the _xmlrpclib accelerator (see the xmlrpclib.py source) uses expat, > with a really fast C layer. judging from Skip's benchmarks, expat is a > bit slower than the py-xmlrpc parser (which is why I asked). I have a feeling py-xmlrpc will slow down a bit when it is internationalized: if (strncmp(*cp, "<int>", 5) == 0) res = decodeInt(cp, ep, lines); else if (strncmp(*cp, "<i4>", 4) == 0) res = decodeI4(cp, ep, lines); .... Paul Prescod From firstalertnews@yahoo.com Tue Oct 2 21:51:59 2001 From: firstalertnews@yahoo.com (firstalertnews@yahoo.com) Date: Tue, 02 Oct 2001 13:51:59 -0700 Subject: [Python-Dev] FIRST ALERT - BTLY: OTCBB Message-ID: <E15oY1F-0001jj-00@mail.python.org> FIRSTALERT'S COMPANY PROFILE FOR THE WEEK OF OCTOBER 1st BENTLEY COMMUNICATIONS CORP: (OTC BB: BTLY) Good day Investors! After the last two weeks of trading everyone is looking for new investments. This week's company profile highlights Bentley Communications Corp. (OTC BB: BTLY). Bentley brings consumers into the world of cutting-edge international gaming websites. Through proprietary technology and advantageous partnership agreements, Bentley operates licensed and fully-insured online gaming websites and sublicenses I-gaming software to businesses worldwide. The Company's flagship I-property, SunriseCityCasino, offers games such as Blackjack, Roulette, Slots, Video Poker and provides users instant access to online betting lines for major sporting events worldwide. Bentley is poised to capture increasing share of the robust Internet sector of I-gaming. By 2002, the global I-gaming market is projected to exceed $1 billion and increase dramatically to more than $9.8 billion by 2005 (source: Merrill Lynch). Bentley targets its websites and software for the world's most popular I-gaming regions, including the United States, Asia and Australia. The Company's growth strategies are directed by Gordon Lee, a polished entrepreneur who has founded several successful emerging-growth companies. Under his leadership, Bentley is poised to enhance its market capitalization substantially over the next 24-months. CORPORATE SUMMARY Bentley Communications Corp. (OTC BB: BTLY) specializes in operating Internet wagering sites. The Company is licensed in both the Commonwealth of Dominica and Antigua to operate sites inclusive of sportsbook, horseracing and bingo. Bentley also sublicenses proprietary technology and software that enable clients to operate their own online gaming sites under the Bentley license. The Company recently announced the launch of a world-class online gaming site SunriseCityCasino.com. Fully licensed and insured, Sunrise offers more than 20 casino games and features a sportsbook with real-time betting lines on worldwide events. Bentley is embarking upon an aggressive growth strategy that comprises geographical expansion into Europe and Asia and new product lines, including the selling of international lottery tickets online. Bentley is also prepared to benefit from new legislation in the state of California. Effective January 1, 2002, telephone and Internet betting on horse racing will be legal in California. Bentley is developing the site offtrackbettingonline.com for launch on 1/2/02, in order to capitalize on this new market in the United States. With numerous revenue streams in place (online casino and sportsbook betting; sublicensing set-up and maintenance fees) and substantial opportunities to grow revenues and profits in the near-term, Bentley is poised to become a leader in the booming online international gaming market. COMPANY DATA Company Name: Bentley Comm Corp (BTLY) Exchange: OTC BB Shares Outstanding: 26,660,000 Market Cap: 2.4 Million 52-Week High: 0.875 on Wednesday, September 27, 2000 52-Week Low: 0.039 on Tuesday, July 31, 2001 Average Price: 0.0990 (50-day) 0.1337 (200-day) Average Volume: 110,100 (50-day) 73,600 (200-day) Fiscal Year End: June 30 Market Makers: 17 INDUSTRY STATISTICS According to Merrill Lynch, the number of online bettors is expected to increase dramatically from 12 million in 2001 to 81 million in 2005, translating to a market size of $769 million and $9.8 billion respectively. Bentley's primary offerings-Internet casinos and sports betting, currently generate the highest gross win (i.e. the amount of cash lost by consumers) in the industry, followed by online lotteries (a developing new product line for Bentley). Bentley's distinct competitive advantages, leading-edge security and fraud protection; a diversified product mix; turn-key proprietary software and technologies and strategic partnerships, will enable the Company to capture increasing share of this dynamic international marketplace. HIGHLIGHTS High-Growth Market. The worldwide Internet gaming market is expected to grow well over 450% over the next two years. Bentley is well-positioned to capitalize on this high-growth sector through the roll-out of several new product lines and regional expansion worldwide. Strategic Partnerships. Bentley has aligned itself with leading technology, banking and security firms to create world-class I-gaming software and Internet websites The Company is associated with the largest e-mail marketing company in the world, Salesmation and has an exclusive agreement with First International Bank of Dominica to provide real-timesecure online financial transactions. Sales & Profitability. Online casinos and sports betting are key revenue generators for Bentley. Additional revenues are generated from sub licensing fees, ranging from $30-50K and the management and maintenance payments of 10% of clients' net revenues. As new products are rolled out, Bentley anticipates to exceed 100% annual revenue growth and achieve profitability during 2001 and beyond. Proven Management. Bentley President and CEO Gordon Lee has a 20-year track record of directing private and public start-ups into successfully operating companies. Mr. Lee has held executive posts including founder, partner, officer, and director at companies such as USA Video Corporation (OTC BB: USVO), Startek.com Inc.(OTC BB: STEK), Future Media Technologies (OTC: FMTF) and American IDC Corp. (OTC BB: ACNI). Upside Opportunity. Bentley is implementing a multifold growth plan that includes international expansion into the fastest growing I-gaming region of Asia; roll-out of new, higher margin product lines such online lottery sales and the launch of additional websites that capitalize on I-gaming growth trends. These efforts should lead to accelerating revenues and increasing shareholder value. EXPERIENCED LEADERSHIP Gordon Lee, president and CEO of Bentley Communications Corp, has been involved with emerging growth companies in the private and public arena for more than 20 years. His astute business acumen has been applied toward a variety of industries, including hi-tech, communications, e-commerce, mining and oil and gas and he has been featured in more than 200 influential media publications. TO OPT OT PLEASE VISIT THE FOLLOWING SITE: http://65.12.58.131/optout.html DISCLAIMER This report is provided as an information service only, and the statements and opinions in this report should not be construed as an offer or solicitation to buy or sell any security. We accept no liability for any loss arising from an investor's reliance on or use of this report. Please do your own due diligence and research before making any investment decision. An investment in the company outlined herein is highly speculative and should not be considered unless you can afford a complete loss of your investment. Please, always invest with caution and within your means. You can find a great resource for investor education at the U.S. Securities and Exchange's website at http://www.sec.gov/investor.shtml. Also, please see the SEC filings of the company profiled herein, including the company's most recent annual and quarterly reports. EquityThunder Investment Corporation received 50,000 free trading shares of Bentley Communications (BTLY) from a third party in exchange for the drafting and distribution of this report. EquityThunder Investment Corporation intends to sell its shares. From guido@python.org Wed Oct 3 00:13:00 2001 From: guido@python.org (Guido van Rossum) Date: Tue, 02 Oct 2001 19:13:00 -0400 Subject: [Python-Dev] Demo/dns In-Reply-To: Your message of "Tue, 02 Oct 2001 14:51:14 PDT." <20011002145114.A20139@glacier.arctrix.com> References: <200110022009.f92K9p301782@mira.informatik.hu-berlin.de> <200110022018.f92KIxV29996@odiug.digicool.com> <20011002145114.A20139@glacier.arctrix.com> Message-ID: <200110022313.TAA18864@cj20424-a.reston1.va.home.com> > Guido van Rossum wrote: > > It's unclear where to leave a pointer -- a Demo/dns directory without > > code feels a little strange. > > Misc/NEWS? > > Neil Good idea. Done. :-) --Guido van Rossum (home page: http://www.python.org/~guido/) From skip@pobox.com (Skip Montanaro) Wed Oct 3 01:33:59 2001 From: skip@pobox.com (Skip Montanaro) (Skip Montanaro) Date: Tue, 2 Oct 2001 19:33:59 -0500 Subject: [Python-Dev] Performance of various marshallers In-Reply-To: <3BBA3C95.1CC094CC@ActiveState.com> References: <15289.7206.266140.820041@beluga.mojam.com> <15289.10654.162126.780312@grendel.zope.com> <03aa01c14b15$e329ac00$b3fa42d5@hagrid> <15289.45881.920165.427067@beluga.mojam.com> <3BBA08C5.240EB4BC@ActiveState.com> <15290.3605.641031.46951@beluga.mojam.com> <3BBA271B.63B07CEF@ActiveState.com> <15290.14155.432881.855383@beluga.mojam.com> <3BBA3C95.1CC094CC@ActiveState.com> Message-ID: <15290.23799.884095.945168@beluga.mojam.com> >> That's precisely why py-xmlrpc is faster. Should it behave some >> other way? I don't think there is another XML-RPC parser out there >> that is available from Python but that doesn't use Python. Paul> Okay, so we agree that the fast part is probably not so much the Paul> parser but the handing of data to Python. So why rewrite a parser? Paul> Nothing requires an Expat-using XML-RPC implementation to call Paul> back into Python for every element. It can collect the results in Paul> C and then call Python when it has values. You're asking the wrong person. Shilad will be the only person who can describe his motivations. We happen to work in the same building, but we don't work for the same company. That's a coincidence about on par with the chances of winning the Powerball lottery. We never met each other formally until about a week ago. Not trying to put words in his mouth, but my guess would be that he was not approaching it as an XML problem, but as a parsing problem. >> I don't understand see how you can't make that connection. XML-RPC >> has a fixed vocabulary and never needs to look at intermediate >> results. Paul> Let me suggest an analogy. Someone writes "CGIPython". It uses a Paul> specially optimized parser designed for parsing only Python CGI Paul> scripts. Do you think it would run much faster than the regular Paul> Python parser? Bad analogy. CGI scripts can contain the entire realm of "stuff" that goes into any other Python program. XML-RPC encodings can't contain arbitrary XML tags or attributes. A better analogy would have been (Martin's I think) hypothetical Swallow - a subset of Python that could be efficiently compiled. Paul> I don't personally see much benefit using XML if you don't adhere Paul> to the XML spec. Just perusing the code quickly I believe I've Paul> found a few bugs that it would not have had if it built on Expat Paul> or some other XML parser. Paul, you have to stop looking at XML-RPC with your Elton John-style XML-colored glasses. XML-RPC is not meant to be some sort of highly structured hierarchical data representation that you can sniff around in with arbitrary XML tools of one sort or another. That its on-the-wire representation happens to be XML is almost ridiculously unimportant. Dave Winer created an RPC tool that used XML at about the same time every computer journalist was wetting their pants every time they heard the letters X-M-L. Many implementations were able to leverage existing XML parsing tools to get going quickly, and Dave got some well-deserved publicity that he and XML-RPC wouldn't have gotten if he'd chosen some other serliazation format like Pickle, or invented something new. Next step: make it go faster. Can that be done with standard XML tools? Yeah, I'm sure it can be. Not everybody approaches the problem with the same background you have though. Paul> 1. It doesn't handle ? syntax. Paul> 2. It doesn't handle <methodCall > (extra whitespace) Paul> 3. I strongly suspect it won't handle comments in the XML. Paul> 4. It won't handle the mandatory UTF-16 encoding from XML Paul> 5. It won't handle CDATA sections. Fine. I'm sure Shilad appreciates the input. I think your approach to bug detection and reporting could have been a bit less heavy handed. As for handling things like CDATA, UTF-16 and extra whitespace after tag names, I suspect some other XML-RPC packages would exhibit similar problems if they were exposed to a standards-toting XML gunslinger like yourself. That it's not a problem in practice is probably because the set of XML-RPC encoding and decoding software is fairly small and that the stuff that encodes into XML-RPC is fairly well-behaved. XML-RPC's widespread availability and practical interoperability (the XML-RPC website lists 48 implementations) probably owes more to the cooperative nature of the people involved than the purity of the parsers. Skip From skip@pobox.com (Skip Montanaro) Wed Oct 3 01:41:04 2001 From: skip@pobox.com (Skip Montanaro) (Skip Montanaro) Date: Tue, 2 Oct 2001 19:41:04 -0500 Subject: [Python-Dev] Performance of various marshallers In-Reply-To: <3BBA3D59.441625E9@ActiveState.com> References: <15289.7206.266140.820041@beluga.mojam.com> <15289.10654.162126.780312@grendel.zope.com> <03aa01c14b15$e329ac00$b3fa42d5@hagrid> <15289.45881.920165.427067@beluga.mojam.com> <3BBA08C5.240EB4BC@ActiveState.com> <15290.3605.641031.46951@beluga.mojam.com> <3BBA271B.63B07CEF@ActiveState.com> <050d01c14b90$213834b0$b3fa42d5@hagrid> <3BBA3D59.441625E9@ActiveState.com> Message-ID: <15290.24224.866683.240585@beluga.mojam.com> Paul> I have a feeling py-xmlrpc will slow down a bit when it is Paul> internationalized: Paul> if (strncmp(*cp, "<int>", 5) == 0) Paul> res = decodeInt(cp, ep, lines); Paul> else if (strncmp(*cp, "<i4>", 4) == 0) Paul> res = decodeI4(cp, ep, lines); Paul> .... Paul, If you want to find and fix bugs in py-xmlrpc or help the author improve the quality of his tools, please send your reports directly to Shilad Sen (shilad@sourcelight.com). Py-xmlrpc has nothing to do with the Python core. I apologize for even including it in the table I posted. Shilad didn't deserve any of the bad press you've given him here. Sending snickering notes to python-dev about the code is not helpful, and only serves to lessen the value I place on your other opinions. Skip From paul@ActiveState.com Wed Oct 3 02:11:47 2001 From: paul@ActiveState.com (Paul Prescod) Date: Tue, 02 Oct 2001 18:11:47 -0700 Subject: [Python-Dev] Performance of various marshallers References: <15289.7206.266140.820041@beluga.mojam.com> <15289.10654.162126.780312@grendel.zope.com> <03aa01c14b15$e329ac00$b3fa42d5@hagrid> <15289.45881.920165.427067@beluga.mojam.com> <3BBA08C5.240EB4BC@ActiveState.com> <15290.3605.641031.46951@beluga.mojam.com> <3BBA271B.63B07CEF@ActiveState.com> <15290.14155.432881.855383@beluga.mojam.com> <3BBA3C95.1CC094CC@ActiveState.com> <15290.23799.884095.945168@beluga.mojam.com> Message-ID: <3BBA65D3.3AE8FAA6@ActiveState.com> Skip Montanaro wrote: > >... > > Bad analogy. CGI scripts can contain the entire realm of "stuff" that goes > into any other Python program. XML-RPC encodings can't contain arbitrary > XML tags or attributes. A better analogy would have been (Martin's I think) > hypothetical Swallow - a subset of Python that could be efficiently > compiled. But there is no evidence that this subset of XML can be more efficiently parsed than any other. XML parsing consists primarily of recognizing angle brackets and a few other characters, and passing around some extra data. Any performance loss from a "full" XML parser will shrink as people submit bug reports that require a "simplified" XML parser to conform to the XML spec (Unicode, CDATA, etc.). I strongly agree that a dedicated C-written XML-RPC implementation can be faster than one written based on Python and Expat. I haven't yet seen evidence that you can both conform with the standards and get much of a speedup over one that is built on a fast XML Parser such as Eric Kidd's XML-RPC C or xmlrpc-epi (both on SourceForge). >... > Paul, you have to stop looking at XML-RPC with your Elton John-style > XML-colored glasses. XML-RPC is not meant to be some sort of highly > structured hierarchical data representation that you can sniff around in > with arbitrary XML tools of one sort or another. That its on-the-wire > representation happens to be XML is almost ridiculously unimportant. XML-RPC uses XML for exactly the same reason every other application of XML uses XML. Precisely so that you will not have to write yet another parser for it. That's the central reason *for* XML. That's the only advantage XML has over cPickle -- that you can be sure whatever language you have, it will have an XML parser available built in. > Fine. I'm sure Shilad appreciates the input. I think your > approach to bug detection and reporting could have been a bit > less heavy handed. I'm not trying to embarrass Shilad. The software isn't at 1.0 yet. Maybe he hasn't got around to choosing an XML parser. I'm trying to point out (more to you, than to him!) that there is a good reason to build on the work other people have done. If pyxmlrpc is faster today it is probably because it doesn't conform to the specs. When it does conform, it won't be faster anymore. > As for handling things like CDATA, UTF-16 and extra whitespace after tag > names, I suspect some other XML-RPC packages would exhibit similar problems > if they were exposed to a standards-toting XML gunslinger like yourself. > That it's not a problem in practice is probably because the set of XML-RPC > encoding and decoding software is fairly small and that the stuff that > encodes into XML-RPC is fairly well-behaved. Every XML-RPC implementation I have ever used (Python, Perl, C, C++, PHP) is based upon one pure XML parser or another. Most use Expat. Paul Prescod From shilad@sourcelight.com Wed Oct 3 02:28:53 2001 From: shilad@sourcelight.com (Shilad Sen) Date: Tue, 2 Oct 2001 20:28:53 -0500 (CDT) Subject: [Python-Dev] Performance of various marshallers In-Reply-To: <15290.23799.884095.945168@beluga.mojam.com> from "Skip Montanaro" at Oct 02, 2001 07:33:59 PM Message-ID: <200110030128.f931Sr005236@jericho.sourcelight.com> Skip has been kind enough to copy me on the bulk of correspondence regarding py-xmlrpc versus other xmlrpc parsing options. py-xmlrpc began as a short hack to accomplish specific things that xmlrpclib couldn't easily accomodate. I used a hand build parser because I thought it would be fun and easy (it was!). Paul, you are correct in that my library doesn't support the 5 items you mentioned. I am aware of these, but they are actually not officially supported by the spec either. XML-RPC is a bit strange in that the spec does not allow or require true XML. My library has been adopted far more than I would have guessed, and I have had many questions about things like SSL support (which is not up to spec either). As a result, I am almost finished with a rewrite that has all the transport and protocol components nicely split up. I have on my list of todo's switching the hand coded parser to expat. My own parser works just fine, though, and I haven't had any complaints so that is relatively low on the list. My library is certainly not as flexible as xmlrpclib in it's current format. I'm hoping that the rewrite will move it to a nice place in the performance / flexibility spectrum. As a side effect, it will have a nice extensible standalone HTTP client and server that offers better performance for people who really need it. I am perfectly aware of py-xmlrpc's shortcomings. On the other hand it is exactly what the app we use needs, and I would be surprised if there aren't others who have similar needs. My hope is that with the next major release, the library will move a bit closer to a place that suits people like Paul. Meanwhile, it works nicely for applications where performance requirements are absolutely critical. Shilad Sen > > >> That's precisely why py-xmlrpc is faster. Should it behave some > >> other way? I don't think there is another XML-RPC parser out there > >> that is available from Python but that doesn't use Python. > > Paul> Okay, so we agree that the fast part is probably not so much the > Paul> parser but the handing of data to Python. So why rewrite a parser? > Paul> Nothing requires an Expat-using XML-RPC implementation to call > Paul> back into Python for every element. It can collect the results in > Paul> C and then call Python when it has values. > > You're asking the wrong person. Shilad will be the only person who can > describe his motivations. We happen to work in the same building, but we > don't work for the same company. That's a coincidence about on par with the > chances of winning the Powerball lottery. We never met each other formally > until about a week ago. Not trying to put words in his mouth, but my guess > would be that he was not approaching it as an XML problem, but as a parsing > problem. > > >> I don't understand see how you can't make that connection. XML-RPC > >> has a fixed vocabulary and never needs to look at intermediate > >> results. > > Paul> Let me suggest an analogy. Someone writes "CGIPython". It uses a > Paul> specially optimized parser designed for parsing only Python CGI > Paul> scripts. Do you think it would run much faster than the regular > Paul> Python parser? > > Bad analogy. CGI scripts can contain the entire realm of "stuff" that goes > into any other Python program. XML-RPC encodings can't contain arbitrary > XML tags or attributes. A better analogy would have been (Martin's I think) > hypothetical Swallow - a subset of Python that could be efficiently > compiled. > > Paul> I don't personally see much benefit using XML if you don't adhere > Paul> to the XML spec. Just perusing the code quickly I believe I've > Paul> found a few bugs that it would not have had if it built on Expat > Paul> or some other XML parser. > > Paul, you have to stop looking at XML-RPC with your Elton John-style > XML-colored glasses. XML-RPC is not meant to be some sort of highly > structured hierarchical data representation that you can sniff around in > with arbitrary XML tools of one sort or another. That its on-the-wire > representation happens to be XML is almost ridiculously unimportant. Dave > Winer created an RPC tool that used XML at about the same time every > computer journalist was wetting their pants every time they heard the > letters X-M-L. Many implementations were able to leverage existing XML > parsing tools to get going quickly, and Dave got some well-deserved > publicity that he and XML-RPC wouldn't have gotten if he'd chosen some other > serliazation format like Pickle, or invented something new. Next step: make > it go faster. Can that be done with standard XML tools? Yeah, I'm sure it > can be. Not everybody approaches the problem with the same background you > have though. > > Paul> 1. It doesn't handle ? syntax. > > Paul> 2. It doesn't handle <methodCall > (extra whitespace) > > Paul> 3. I strongly suspect it won't handle comments in the XML. > > Paul> 4. It won't handle the mandatory UTF-16 encoding from XML > > Paul> 5. It won't handle CDATA sections. > > Fine. I'm sure Shilad appreciates the input. I think your approach to bug > detection and reporting could have been a bit less heavy handed. > > As for handling things like CDATA, UTF-16 and extra whitespace after tag > names, I suspect some other XML-RPC packages would exhibit similar problems > if they were exposed to a standards-toting XML gunslinger like yourself. > That it's not a problem in practice is probably because the set of XML-RPC > encoding and decoding software is fairly small and that the stuff that > encodes into XML-RPC is fairly well-behaved. > > XML-RPC's widespread availability and practical interoperability (the > XML-RPC website lists 48 implementations) probably owes more to the > cooperative nature of the people involved than the purity of the parsers. > > Skip > From barry@zope.com Wed Oct 3 02:39:04 2001 From: barry@zope.com (Barry A. Warsaw) Date: Tue, 2 Oct 2001 21:39:04 -0400 Subject: [Python-Dev] Cookie.py and `:' in the key Message-ID: <15290.27704.19609.960912@anthem.wooz.org> In Mailman, I use a version of Cookie.py written by Timothy dated from 1998. I'm now trying to see if I can get rid of my independent copy and just use Cookie.py in the Python 2.x standard library. I've hit a snag; in Mailman's copy, it is legal to have a `:' in the key name, but in Python's Cookie.py it isn't: -------------------- snip snip -------------------- % python Python 2.1.1 (#1, Aug 31 2001, 17:07:00) [GCC egcs-2.91.66 19990314/Linux (egcs-1.1.2 release)] on linux2 Type "copyright", "credits" or "license" for more information. >>> import Cookie >>> c = Cookie.Cookie() >>> c['foo:bar'] = 'hello' Traceback (most recent call last): File "<stdin>", line 1, in ? File "/usr/local/lib/python2.1/Cookie.py", line 583, in __setitem__ self.__set(key, rval, cval) File "/usr/local/lib/python2.1/Cookie.py", line 576, in __set M.set(key, real_value, coded_value) File "/usr/local/lib/python2.1/Cookie.py", line 456, in set raise CookieError("Illegal key value: %s" % key) Cookie.CookieError: Illegal key value: foo:bar >>> % PYTHONPATH=/path/to/mailman/misc python Python 2.1.1 (#1, Aug 31 2001, 17:07:00) [GCC egcs-2.91.66 19990314/Linux (egcs-1.1.2 release)] on linux2 Type "copyright", "credits" or "license" for more information. >>> import Cookie >>> c = Cookie.Cookie() >>> c['foo:bar'] = 'hello' >>> print c Set-Cookie: foo:bar=hello; >>> -------------------- snip snip -------------------- I don't see any reason why `:' shouldn't be allowed in Set-Cookie: value, but maybe I'm missing something in the RFCs. This patch fixes the problem but perhaps not in the right way. Comments? -Barry -------------------- snip snip -------------------- Index: Cookie.py =================================================================== RCS file: /cvsroot/python/python/dist/src/Lib/Cookie.py,v retrieving revision 1.11 diff -u -r1.11 Cookie.py --- Cookie.py 2001/08/02 07:15:29 1.11 +++ Cookie.py 2001/10/03 01:38:39 @@ -249,7 +249,7 @@ # _LegalChars is the list of chars which don't require "'s # _Translator hash-table for fast quoting # -_LegalChars = string.ascii_letters + string.digits + "!#$%&'*+-.^_`|~" +_LegalChars = string.ascii_letters + string.digits + "!#$%&'*+-.^_`|~:" _Translator = { '\000' : '\\000', '\001' : '\\001', '\002' : '\\002', '\003' : '\\003', '\004' : '\\004', '\005' : '\\005', From paul@ActiveState.com Wed Oct 3 02:47:06 2001 From: paul@ActiveState.com (Paul Prescod) Date: Tue, 02 Oct 2001 18:47:06 -0700 Subject: [Python-Dev] Performance of various marshallers References: <15289.7206.266140.820041@beluga.mojam.com> <15289.10654.162126.780312@grendel.zope.com> <03aa01c14b15$e329ac00$b3fa42d5@hagrid> <15289.45881.920165.427067@beluga.mojam.com> <3BBA08C5.240EB4BC@ActiveState.com> <15290.3605.641031.46951@beluga.mojam.com> <3BBA271B.63B07CEF@ActiveState.com> <050d01c14b90$213834b0$b3fa42d5@hagrid> <3BBA3D59.441625E9@ActiveState.com> <15290.24224.866683.240585@beluga.mojam.com> Message-ID: <3BBA6E1A.363F4E79@ActiveState.com> I apologize if I embarassed Shilad. I don't know him so I don't know how he will take a public critique of his code. For all I know, he agrees with me and merely hasn't got around to adding in an XML parser. On the one hand, I can see how it would be nicer to discuss it directly with you and him, but on the other, it is a real technical issue that deserves public discussion. I felt (and feel) that you've made a technical mistake in attributing py-xmlrpc's speed to its having a fixed tagset and I only posted code to demonstrate where the real speedup comes from. I've spent my whole life working around bugs in hand-rolled XML (and SGML) parsers that are supposed to be faster than general ones but end up not being so. I react almost as intemperately when someone tells me that their app embeds a new scripting language that they invented over the weekend. Although I do think that the current parsing approach taken in py-xmlrpc is flawed, I do think that the overall idea is good. It makes sense to parse XML-RPC purely in C without using Python callbacks. Paul Prescod From paul@ActiveState.com Wed Oct 3 02:57:35 2001 From: paul@ActiveState.com (Paul Prescod) Date: Tue, 02 Oct 2001 18:57:35 -0700 Subject: [Python-Dev] Performance of various marshallers References: <200110030128.f931Sr005236@jericho.sourcelight.com> Message-ID: <3BBA708F.D8554BBC@ActiveState.com> Shilad Sen wrote: > > Skip has been kind enough to copy me on the bulk of correspondence > regarding py-xmlrpc versus other xmlrpc parsing options. Thanks for your good-natured response. >... > Paul, you are correct in that my library doesn't support the 5 items you > mentioned. I am aware of these, but they are actually not officially > supported by the spec either. XML-RPC is a bit strange in that the spec > does not allow or require true XML. I think that if a spec claims to be based on XML and does not explicitly disclaim support for built-in XML features, then it allows them. For instance if it doesn't say that C syntax is illegal, then there is no reason to believe it is. > My library has been adopted far more than I would have guessed, and I > have had many questions about things like SSL support (which is not up > to spec either). As a result, I am almost finished with a rewrite that > has all the transport and protocol components nicely split up. I have > on my list of todo's switching the hand coded parser to expat. My own > parser works just fine, though, and I haven't had any complaints > so that is relatively low on the list. That's fine with me. If your simplified parser turns out to be significantly faster than Expat (too early to say) then you could even keep it around as an option when the client and the server are both known to be using the same subset of XML. > ... My hope is that with the next > major release, the library will move a bit closer to a place that suits > people like Paul. Meanwhile, it works nicely for applications where > performance requirements are absolutely critical. Did you consider wrapping one of the existing XML-RPC libraries written in C? When we needed a reentrant XML-RPC library for PHP, we wrapped Eric Kidd's xmlrpc-c. Paul Prescod From shilad@sourcelight.com Wed Oct 3 03:12:43 2001 From: shilad@sourcelight.com (Shilad Sen) Date: Tue, 2 Oct 2001 21:12:43 -0500 (CDT) Subject: [Python-Dev] Performance of various marshallers In-Reply-To: <3BBA708F.D8554BBC@ActiveState.com> from "Paul Prescod" at Oct 02, 2001 06:57:35 PM Message-ID: <200110030212.f932Chn05354@jericho.sourcelight.com> >From Paul: > > That's fine with me. If your simplified parser turns out to be > significantly faster than Expat (too early to say) then you could even > keep it around as an option when the client and the server are both > known to be using the same subset of XML. > That's exactly what I intended to do. > > Did you consider wrapping one of the existing XML-RPC libraries written > in C? When we needed a reentrant XML-RPC library for PHP, we wrapped > Eric Kidd's xmlrpc-c. > I actually seriously considered using Eric Kidd's library. We have a need for an event model that would have been pretty painful to support through the w3c libraries, and that is what made me decide to use my own. My event system is very similar to medusa's, while w3c's system is quite a bit more heavyweight. After spending some time with the w3c libraries, I also decided I really didn't feel great about the general coding style of the libraries. Thats my own personal preference. I could definitely be persuaded otherwise. Shilad From skip@pobox.com (Skip Montanaro) Wed Oct 3 03:16:40 2001 From: skip@pobox.com (Skip Montanaro) (Skip Montanaro) Date: Tue, 2 Oct 2001 21:16:40 -0500 Subject: [Python-Dev] Performance of various marshallers In-Reply-To: <3BBA65D3.3AE8FAA6@ActiveState.com> References: <15289.7206.266140.820041@beluga.mojam.com> <15289.10654.162126.780312@grendel.zope.com> <03aa01c14b15$e329ac00$b3fa42d5@hagrid> <15289.45881.920165.427067@beluga.mojam.com> <3BBA08C5.240EB4BC@ActiveState.com> <15290.3605.641031.46951@beluga.mojam.com> <3BBA271B.63B07CEF@ActiveState.com> <15290.14155.432881.855383@beluga.mojam.com> <3BBA3C95.1CC094CC@ActiveState.com> <15290.23799.884095.945168@beluga.mojam.com> <3BBA65D3.3AE8FAA6@ActiveState.com> Message-ID: <15290.29960.899139.647577@beluga.mojam.com> >> Paul, you have to stop looking at XML-RPC with your Elton John-style >> XML-colored glasses. XML-RPC is not meant to be some sort of highly >> structured hierarchical data representation that you can sniff around >> in with arbitrary XML tools of one sort or another. That its >> on-the-wire representation happens to be XML is almost ridiculously >> unimportant. Paul> XML-RPC uses XML for exactly the same reason every other Paul> application of XML uses XML. I disagree with that. Lots of applications use XML because it's got that pants-wetting capability I described earlier. >> Fine. I'm sure Shilad appreciates the input. I think your approach >> to bug detection and reporting could have been a bit less heavy >> handed. Paul> I'm not trying to embarrass Shilad. The software isn't at 1.0 Paul> yet. Maybe he hasn't got around to choosing an XML parser. Or maybe he has a different set of constraints than you. Paul> I'm trying to point out (more to you, than to him!) that there is Paul> a good reason to build on the work other people have done. If Paul> pyxmlrpc is faster today it is probably because it doesn't conform Paul> to the specs. When it does conform, it won't be faster anymore. Why point this out to me? I am essentially just an XML-RPC user, not an implementer. I happen to be interested in making my XML-RPC-using code run faster. If I have to make some sacrifices I could care less, as long as my clients and my servers can talk to one another. >> As for handling things like CDATA, UTF-16 and extra whitespace after >> tag names, I suspect some other XML-RPC packages would exhibit >> similar problems if they were exposed to a standards-toting XML >> gunslinger like yourself. That it's not a problem in practice is >> probably because the set of XML-RPC encoding and decoding software is >> fairly small and that the stuff that encodes into XML-RPC is fairly >> well-behaved. Paul> Every XML-RPC implementation I have ever used (Python, Perl, C, Paul> C++, PHP) is based upon one pure XML parser or another. Most use Paul> Expat. Oh well. S From skip@pobox.com (Skip Montanaro) Wed Oct 3 03:23:31 2001 From: skip@pobox.com (Skip Montanaro) (Skip Montanaro) Date: Tue, 2 Oct 2001 21:23:31 -0500 Subject: [Python-Dev] Performance of various marshallers In-Reply-To: <3BBA708F.D8554BBC@ActiveState.com> References: <200110030128.f931Sr005236@jericho.sourcelight.com> <3BBA708F.D8554BBC@ActiveState.com> Message-ID: <15290.30371.513079.764350@beluga.mojam.com> Paul> I think that if a spec claims to be based on XML and does not Paul> explicitly disclaim support for built-in XML features, then it Paul> allows them. For instance if it doesn't say that C syntax is Paul> illegal, then there is no reason to believe it is. Paul, You probably know as well as anyone that the one and only person you should talking to about XML-RPC and its XML compliance (or lack thereof) is Dave Winer. Feel free to read through the archives of the xmlrpc@yahoogroups.com mailing list if you haven't already. If you can move Dave from his current position, more power to you. You'll do something that many other people have been incapable of doing. I'm done with this topic. It's gotten way too far from python-dev-related topics. Probably should have cut it out of the cc list awhile back. Skip From guido@python.org Wed Oct 3 04:05:32 2001 From: guido@python.org (Guido van Rossum) Date: Tue, 02 Oct 2001 23:05:32 -0400 Subject: [Python-Dev] Performance of various marshallers In-Reply-To: Your message of "Tue, 02 Oct 2001 21:23:31 CDT." <15290.30371.513079.764350@beluga.mojam.com> References: <200110030128.f931Sr005236@jericho.sourcelight.com> <3BBA708F.D8554BBC@ActiveState.com> <15290.30371.513079.764350@beluga.mojam.com> Message-ID: <200110030305.XAA26130@cj20424-a.reston1.va.home.com> > I'm done with this topic. It's gotten way too far from python-dev-related > topics. Probably should have cut it out of the cc list awhile back. Amen. --Guido van Rossum (home page: http://www.python.org/~guido/) From timo@alum.mit.edu Wed Oct 3 06:18:36 2001 From: timo@alum.mit.edu (Timothy O'Malley) Date: Wed, 3 Oct 2001 01:18:36 -0400 Subject: [Python-Dev] Re: Cookie.py and `:' in the key In-Reply-To: <15290.27704.19609.960912@anthem.wooz.org> Message-ID: <1890D9DE-B7BE-11D5-B9BF-00306586E61E@alum.mit.edu> hola. On Tuesday, October 2, 2001, at 09:39 PM, Barry A. Warsaw wrote: > In Mailman, I use a version of Cookie.py written by Timothy dated from > 1998. I'm now trying to see if I can get rid of my independent > copy and just use Cookie.py in the Python 2.x standard library. According to a very strict reading of the appropriate specifications (RFC 2109 for cookies, which in turn references terms defined in RFC 2068 for HTTP), a colon is not legal in a value unless it is in a quoted value: Many HTTP/1.1 header field values consist of words separated by LWS or special characters. These special characters MUST be in a quoted string to be used within a parameter value. token = 1*<any CHAR except CTLs or tspecials> tspecials = "(" | ")" | "<" | ">" | "@" | "," | ";" | ":" | "\" | <"> | "/" | "[" | "]" | "?" | "=" | "{" | "}" | SP | HT Even so, I think I agree that a strict interpretation isn't very useful in practice. In this case, for instance, the intended value is clear and obvious. I've gone back and forth on this -- should the implementation be true to the spec or should it follow its own rules for clear and obvious? Should people desire the "clear and obvious" over the "strict interpretation", I think your fix is dead on. TimO From skip@pobox.com (Skip Montanaro) Wed Oct 3 08:40:28 2001 From: skip@pobox.com (Skip Montanaro) (Skip Montanaro) Date: Wed, 3 Oct 2001 02:40:28 -0500 Subject: [Python-Dev] dummy __del__ in SocketServer.BaseRequestHandler Message-ID: <15290.49388.86639.812781@beluga.mojam.com> Given that the presence of a __del__ method keeps the garbage collector from reclaiming cyclic garbage, should SocketServer.BaseRequestHandler define a __del__ method that just executes "pass"? Commenting it out seems to have gotten rid of some "uncollectable" messages from the garbage collector. Every other __del__ method in the top level of the standard library actually does something. Skip From guido@python.org Wed Oct 3 12:53:42 2001 From: guido@python.org (Guido van Rossum) Date: Wed, 03 Oct 2001 07:53:42 -0400 Subject: [Python-Dev] dummy __del__ in SocketServer.BaseRequestHandler In-Reply-To: Your message of "Wed, 03 Oct 2001 02:40:28 CDT." <15290.49388.86639.812781@beluga.mojam.com> References: <15290.49388.86639.812781@beluga.mojam.com> Message-ID: <200110031153.HAA27628@cj20424-a.reston1.va.home.com> > Given that the presence of a __del__ method keeps the garbage collector from > reclaiming cyclic garbage, should SocketServer.BaseRequestHandler define a > __del__ method that just executes "pass"? Commenting it out seems to have > gotten rid of some "uncollectable" messages from the garbage collector. > Every other __del__ method in the top level of the standard library actually > does something. +1. --Guido van Rossum (home page: http://www.python.org/~guido/) From skip@pobox.com (Skip Montanaro) Wed Oct 3 20:18:05 2001 From: skip@pobox.com (Skip Montanaro) (Skip Montanaro) Date: Wed, 3 Oct 2001 14:18:05 -0500 Subject: [Python-Dev] a bit confused about the new type/class stuff Message-ID: <15291.25709.109749.875704@beluga.mojam.com> I have a class hierarchy I'm trying to migrate to 2.2 using the latest stuff from CVS on both the Gtk and Python sides of the fence. Conceptually its skeleton looks like class Object: ... class Widget(Object): ... class Button(Widget): ... (It's a bunch of wrappers around Gtk widgets. The wrappers happen to use delegation instead of instantiation, so my Button class is not subclassed from gtk.Button. In particular, it's *not* one of the new subclassable types.) At one point in my code I test to see if one of my Button instances is in a list of Widget instances and I get a TypeError: TypeError: Button.__cmp__(x,y) requires y to be a 'Button', not a 'instance' If I test for b's inclusion in l, even if I draw b from l: >>> l [<Button object (GtkButton) at 0x8293134>, <Button object (GtkButton) at 0x8296ac4>, <Button object (GtkButton) at 0x8397fb4>] >>> b = l[0] >>> b in l Traceback (most recent call last): File "<stdin>", line 1, in ? TypeError: Button.__cmp__(x,y) requires y to be a 'Button', not a 'instance' I get an error. I don't define a __cmp__ method for the Button class or any of its ancestors, but if I print b.__cmp__ something is definitely there: >>> b.__cmp__ <method-wrapper object at 0x83aae78> This was all working using Python 2.1. I haven't changed anything so far in my code other than to adapt to the new PyGtk naming scheme (e.g., gtk.Button vs. gtk.GtkButton) and recompile/install Gtk, PyGtk, Python and friends. In particular, I believe I have a hierarchy of so-called "classic" classes. I'm going to toddle off an reread the various PEPs, but would appreciate an insight or two on where this error is coming from. Thx, -- Skip Montanaro (skip@pobox.com) http://www.mojam.com/ http://www.musi-cal.com/ From skip@pobox.com (Skip Montanaro) Wed Oct 3 21:05:32 2001 From: skip@pobox.com (Skip Montanaro) (Skip Montanaro) Date: Wed, 3 Oct 2001 15:05:32 -0500 Subject: [Python-Dev] Re: [pygtk] a bit confused about the new type/class stuff In-Reply-To: <15291.25709.109749.875704@beluga.mojam.com> References: <15291.25709.109749.875704@beluga.mojam.com> Message-ID: <15291.28556.67625.136475@beluga.mojam.com> ... Skip> (It's a bunch of wrappers around Gtk widgets. The wrappers happen Skip> to use delegation instead of instantiation, so my Button class is Skip> not subclassed from gtk.Button. In particular, it's *not* one of Skip> the new subclassable types.) God, I was up way too late last night. Make that "delegation instead of inheritance". still-stumped-in-evanston-ly, y'rs, Skip From gward@python.net Wed Oct 3 23:13:04 2001 From: gward@python.net (Greg Ward) Date: Wed, 3 Oct 2001 18:13:04 -0400 Subject: [Python-Dev] Making setup.py a little more flexible Message-ID: <20011003181304.A17246@cthulhu.mems-exchange.org> Some months ago, I tried to build the latest Python on an IRIX 6.5 system that has been, shall we say, lightly maintained for some time. (Basically, its /usr/local/ is mounted from an IRIX 5 system, and it's full of libraries that you can't link against with the default compiler flags on IRIX 6.) Unfortunately, Python's setup.py is a bit too rigid: it assumes that it should always add /usr/local/{lib,include} to the lib/include search paths, it assumes it should always search /usr/local/lib (and a few others) for SSL libraries, etc. Worse, it assumes the existence of libssl.a implies the linkability of -lssl -- when /usr/local is broken, this ain't necessarily so. Now I know what the Right Answer is: finish implementing the Distutils "config" command, and use it to properly probe the system for includable header files and linkable libraries, rather than just assuming existence == validity. That would be nice. But I've got a quick 'n easy fix -- the moral equivalent of commenting out the code in PyBuildExt.detect_modules() that makes those blithe assumptions. Namely, I added --without-usrlocal and --without-ssl options to the custom build_ext command used by setup.py. Now, I can build most extensions just fine on this broken IRIX system like this: ./python setup.py build_ext --without-ssl --without-usrlocal I've just uploaded patch #467650 (assigned to Andrew) to implement this: http://sourceforge.net/tracker/index.php?func=detail&aid=467650&group_id=5470&atid=305470 Whaddya think? Good enough bandaid solution? Or should I add a few more things, eg.: --with-ssl-lib=DIR directory to search for OpenSSL libs --with-ssl-inc=DIR directory to search for OpenSSL headers --without-foo don't try to build foo The last one could be repeated for various values of "foo": curses, dbm, bsddb, etc. I don't think that's too necessary; if some foo doesn't build, then it doesn't build and that's that. From my cursory examination of detect_modules(), the _socket/OpenSSL dependency is the only obvious module-killer. (That is, the bogus OpenSSL libs killed the _socket module completely, when all I really need to do is not use OpenSSL on this system.) Greg -- Greg Ward - Python bigot gward@python.net http://starship.python.net/~gward/ I haven't lost my mind; I know exactly where I left it. From gward@python.net Thu Oct 4 02:16:35 2001 From: gward@python.net (Greg Ward) Date: Wed, 3 Oct 2001 21:16:35 -0400 Subject: [Python-Dev] Fixing compiler warnings (bug #445960) Message-ID: <01100321143900.18648@cthulhu> What's the general feeling/policy on fixing very minor problems that cause legitimate compiler warnings with some compilers? ISTR that Tim is adamant about keeping Python warning-free on Windows with MSVC, but what about, say, the compiler that ships with IRIX 6.5? It's not a widely-used platform, but the compiler does catch some genuine, albeit small, problems -- mostly of the "variable set but never used" variety. It looks like most fixes are along the lines of changing this: static int init_builtin(char *name) { struct _inittab *p; PyObject *mod; if ((mod = _PyImport_FindExtension(name, name)) != NULL) return 1; to this: static int init_builtin(char *name) { struct _inittab *p; From guido@python.org Thu Oct 4 02:28:19 2001 From: guido@python.org (Guido van Rossum) Date: Wed, 03 Oct 2001 21:28:19 -0400 Subject: [Python-Dev] a bit confused about the new type/class stuff In-Reply-To: Your message of "Wed, 03 Oct 2001 14:18:05 CDT." <15291.25709.109749.875704@beluga.mojam.com> References: <15291.25709.109749.875704@beluga.mojam.com> Message-ID: <200110040128.VAA32301@cj20424-a.reston1.va.home.com> > I have a class hierarchy I'm trying to migrate to 2.2 using the latest stuff > from CVS on both the Gtk and Python sides of the fence. Conceptually its > skeleton looks like > > class Object: > ... > > class Widget(Object): > ... > > class Button(Widget): > ... > > (It's a bunch of wrappers around Gtk widgets. The wrappers happen to use > delegation instead of instantiation, so my Button class is not subclassed > from gtk.Button. In particular, it's *not* one of the new subclassable > types.) > > At one point in my code I test to see if one of my Button instances is in a > list of Widget instances and I get a TypeError: > > TypeError: Button.__cmp__(x,y) requires y to be a 'Button', not a 'instance' This error comes from wrap_cmpfunc() in typeobject.c. It should only happen when you are somehow comparing new-style objects. > If I test for b's inclusion in l, even if I draw b from l: > > >>> l > [<Button object (GtkButton) at 0x8293134>, <Button object (GtkButton) at 0x8296ac4>, <Button object (GtkButton) at 0x8397fb4>] > >>> b = l[0] > >>> b in l > Traceback (most recent call last): > File "<stdin>", line 1, in ? > TypeError: Button.__cmp__(x,y) requires y to be a 'Button', not a 'instance' > > I get an error. What do you get when you ask for (b == l)? > I don't define a __cmp__ method for the Button class or any of its > ancestors, but if I print b.__cmp__ something is definitely there: > > >>> b.__cmp__ > <method-wrapper object at 0x83aae78> That's truly bizarre. A <method-wrapper object> is an *unbound* __cmp__ method that wraps the tp_compare slot of a built-in type. I have no explanation for how it would show up in your instance. Can you check whether b.__cmp__ is the same as b.__dict__['__cmp__'] or whether it is the same as b.__class__.__cmp__? It should be either one of those. Then, if it's in b.__class__, try to find out in which class's __dict__ it lives. > This was all working using Python 2.1. I haven't changed anything so far in > my code other than to adapt to the new PyGtk naming scheme (e.g., gtk.Button > vs. gtk.GtkButton) and recompile/install Gtk, PyGtk, Python and friends. In > particular, I believe I have a hierarchy of so-called "classic" classes. > > I'm going to toddle off an reread the various PEPs, but would appreciate an > insight or two on where this error is coming from. I have zero experience with Gtk, but I heard somewhere that it uses the Don Beaudry hook. This is enabled in recent 2.2, but it might act slightly different than expected. This is the first thing I'd investigate. Good luck! (If you can mail me step-by-step instructions for reproducing your situation on Linux, including URLs of specific Gtk source tarballs etc., I might play with this a bit...) --Guido van Rossum (home page: http://www.python.org/~guido/) From tim.one@home.com Thu Oct 4 03:04:38 2001 From: tim.one@home.com (Tim Peters) Date: Wed, 3 Oct 2001 22:04:38 -0400 Subject: [Python-Dev] Fixing compiler warnings (bug #445960) In-Reply-To: <01100321143900.18648@cthulhu> Message-ID: <LNBBLJKPBEHFEDALKOLCAEDBLPAA.tim.one@home.com> [Greg Ward] > What's the general feeling/policy on fixing very minor problems > that cause legitimate compiler warnings with some compilers? If they're truly problems, they should be fixed. > ISTR that Tim is adamant about keeping Python warning-free on Windows > with MSVC, Yes, but note that MSVC doesn't complain about non-problems <0.9 wink -- it actually does, but not at the wng level we use there>. > but what about, say, the compiler that ships with IRIX 6.5? Doesn't have to do with the compiler so much as the specifics of what it's complaining about. MSVC and gcc and XXX catch different things, and that's valuable. > It's not a widely-used platform, but the compiler does catch some > genuine, albeit small, problems -- mostly of the "variable set but > never used" variety. It looks like most fixes are along the lines of > changing this: > > static int > init_builtin(char *name) > { > struct _inittab *p; > PyObject *mod; > > if ((mod = _PyImport_FindExtension(name, name)) != NULL) > return 1; > > to this: > > static int > init_builtin(char *name) > { > struct _inittab *p; Seems to be missing a line; I assume if (_PImport_FindExtension(name, name) != NULL) return 1; was intended. I'd make that change, as there's no downside. Examples of changes I wouldn't make: 1. A compiler warning when an __inline function isn't inlined. 2. A compiler complaining about a vrbl possibly used before initialization, when it's instantly obvious that the vrbl is initialized on all paths before use (got a report like that with two of those yesterday -- won't "fix"). 3. Often (but not always) a complaint that an assignment is stomped on before use. This one I'd do something about: a = 1; /* complaint about this */ a = 2; This one I wouldn't: PyObject *result = NULL; /* complaint about this */ /* 300 lines of excruciating code, every one of the 116 paths through which just happens to set result before it's used. */ return result; A special problem with #2 and #3 is that shutting up a compiler that complains about one of them often creates code that another compiler will complain about for the other reason. That's one reason "no warning from any compiler" is impossible to achieve. From skip@pobox.com (Skip Montanaro) Thu Oct 4 03:07:40 2001 From: skip@pobox.com (Skip Montanaro) (Skip Montanaro) Date: Wed, 3 Oct 2001 21:07:40 -0500 Subject: [Python-Dev] a bit confused about the new type/class stuff In-Reply-To: <200110040128.VAA32301@cj20424-a.reston1.va.home.com> References: <15291.25709.109749.875704@beluga.mojam.com> <200110040128.VAA32301@cj20424-a.reston1.va.home.com> Message-ID: <15291.50284.699868.549699@beluga.mojam.com> Guido, Thanks for the various tips. I will check 'em out. Regarding the Don Beaudry hook, PyGtk was using ExtensionClass until recently. James Henstridge has just released the first cut at using your new type/class stuff in the past few days. So if you didn't use it, neither does PyGtk. Guido> I have zero experience with Gtk, but I heard somewhere that it Guido> uses the Don Beaudry hook. This is enabled in recent 2.2, but it Guido> might act slightly different than expected. This is the first Guido> thing I'd investigate. Guido> Good luck! (If you can mail me step-by-step instructions for Guido> reproducing your situation on Linux, including URLs of specific Guido> Gtk source tarballs etc., I might play with this a bit...) This is too new to have tarballs yet (lots of hairballs perhaps...). Everything is coming out of various CVS repositories. If you want to waste a half a day getting all the Gtk build tools and libraries correct, I can set you up, but I suspect you may have better things to do with your time. An alternative is for me to give you ssh access to my laptop. You could be fiddling around with this stuff in a few minutes. Even if it wasn't useful to track my problem down you might like kicking the tires on a pretty large client for your new type/class code. Skip From mal@lemburg.com Thu Oct 4 09:44:28 2001 From: mal@lemburg.com (M.-A. Lemburg) Date: Thu, 04 Oct 2001 10:44:28 +0200 Subject: [Python-Dev] Making setup.py a little more flexible References: <20011003181304.A17246@cthulhu.mems-exchange.org> Message-ID: <3BBC216C.C38AC411@lemburg.com> Greg Ward wrote: > > Some months ago, I tried to build the latest Python on an IRIX 6.5 > system that has been, shall we say, lightly maintained for some time. > (Basically, its /usr/local/ is mounted from an IRIX 5 system, and it's > full of libraries that you can't link against with the default compiler > flags on IRIX 6.) > > Unfortunately, Python's setup.py is a bit too rigid: it assumes that it > should always add /usr/local/{lib,include} to the lib/include search > paths, it assumes it should always search /usr/local/lib (and a few > others) for SSL libraries, etc. Worse, it assumes the existence of > libssl.a implies the linkability of -lssl -- when /usr/local is broken, > this ain't necessarily so. > > Now I know what the Right Answer is: finish implementing the Distutils > "config" command, and use it to properly probe the system for includable > header files and linkable libraries, rather than just assuming existence > == validity. > That would be nice. That would certainly be nice indeed (for other users of distutils as well ;-). > But I've got a quick 'n easy fix -- the moral > equivalent of commenting out the code in PyBuildExt.detect_modules() > that makes those blithe assumptions. Namely, I added --without-usrlocal > and --without-ssl options to the custom build_ext command used by > setup.py. > > Now, I can build most extensions just fine on this broken IRIX system > like this: > ./python setup.py build_ext --without-ssl --without-usrlocal > > I've just uploaded patch #467650 (assigned to Andrew) to implement this: > http://sourceforge.net/tracker/index.php?func=detail&aid=467650&group_id=5470&atid=305470 > > Whaddya think? Good enough bandaid solution? Or should I add a few > more things, eg.: > --with-ssl-lib=DIR directory to search for OpenSSL libs > --with-ssl-inc=DIR directory to search for OpenSSL headers > --without-foo don't try to build foo I think that these options are more in line with what configure scripts tend to offer: if the magic doesn't work, let the user decide. > The last one could be repeated for various values of "foo": curses, dbm, > bsddb, etc. I don't think that's too necessary; if some foo doesn't > build, then it doesn't build and that's that. From my cursory > examination of detect_modules(), the _socket/OpenSSL dependency is the > only obvious module-killer. (That is, the bogus OpenSSL libs killed the > _socket module completely, when all I really need to do is not use > OpenSSL on this system.) -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Consulting & Company: http://www.egenix.com/ Python Software: http://www.lemburg.com/python/ From mal@lemburg.com Thu Oct 4 09:48:01 2001 From: mal@lemburg.com (M.-A. Lemburg) Date: Thu, 04 Oct 2001 10:48:01 +0200 Subject: [Python-Dev] Fixing compiler warnings (bug #445960) References: <01100321143900.18648@cthulhu> Message-ID: <3BBC2241.7BDFB9B1@lemburg.com> Greg Ward wrote: > > What's the general feeling/policy on fixing very minor problems that cause > legitimate compiler warnings with some compilers? ISTR that Tim is adamant > about keeping Python warning-free on Windows with MSVC, but what about, say, > the compiler that ships with IRIX 6.5? It's not a widely-used platform, but > the compiler does catch some genuine, albeit small, problems -- mostly of the > "variable set but never used" variety. > > It looks like most fixes are along the lines of changing this: > > static int > init_builtin(char *name) > { > struct _inittab *p; > PyObject *mod; > > if ((mod = _PyImport_FindExtension(name, name)) != NULL) > return 1; > > to this: > > static int > init_builtin(char *name) > { > struct _inittab *p; > > if (_PyImport_FindExtension(name, name) != NULL) > return 1; If it improves code quality, I'd say that any valid code warning should be considered and fixed (doesn't really matter what the source is ;-). -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Consulting & Company: http://www.egenix.com/ Python Software: http://www.lemburg.com/python/ From guido@python.org Thu Oct 4 11:29:18 2001 From: guido@python.org (Guido van Rossum) Date: Thu, 04 Oct 2001 06:29:18 -0400 Subject: [Python-Dev] a bit confused about the new type/class stuff In-Reply-To: Your message of "Wed, 03 Oct 2001 21:07:40 CDT." <15291.50284.699868.549699@beluga.mojam.com> References: <15291.25709.109749.875704@beluga.mojam.com> <200110040128.VAA32301@cj20424-a.reston1.va.home.com> <15291.50284.699868.549699@beluga.mojam.com> Message-ID: <200110041029.GAA01791@cj20424-a.reston1.va.home.com> > Thanks for the various tips. I will check 'em out. Regarding the Don > Beaudry hook, PyGtk was using ExtensionClass until recently. James > Henstridge has just released the first cut at using your new type/class > stuff in the past few days. So if you didn't use it, neither does PyGtk. Ah, but then it could be that somehow this cutting-edge PyGtk has caused your immediate problem, right? I still take responsibility -- the new type stuff isn't documented enough yet. :-( > This is too new to have tarballs yet (lots of hairballs perhaps...). > Everything is coming out of various CVS repositories. If you want to waste > a half a day getting all the Gtk build tools and libraries correct, I can > set you up, but I suspect you may have better things to do with your time. Yup. > An alternative is for me to give you ssh access to my laptop. You could be > fiddling around with this stuff in a few minutes. Even if it wasn't useful > to track my problem down you might like kicking the tires on a pretty large > client for your new type/class code. If you mail me your phone number in private I'll call you to coordinate this more, when I have time. --Guido van Rossum (home page: http://www.python.org/~guido/) From james@daa.com.au Thu Oct 4 13:26:45 2001 From: james@daa.com.au (James Henstridge) Date: Thu, 4 Oct 2001 20:26:45 +0800 (WST) Subject: [pygtk] Re: [Python-Dev] a bit confused about the new type/class stuff In-Reply-To: <200110041029.GAA01791@cj20424-a.reston1.va.home.com> Message-ID: <Pine.LNX.4.33.0110042015050.7347-100000@james.daa.com.au> On Thu, 4 Oct 2001, Guido van Rossum wrote: > > Thanks for the various tips. I will check 'em out. Regarding the Don > > Beaudry hook, PyGtk was using ExtensionClass until recently. James > > Henstridge has just released the first cut at using your new type/class > > stuff in the past few days. So if you didn't use it, neither does PyGtk. > > Ah, but then it could be that somehow this cutting-edge PyGtk has > caused your immediate problem, right? I still take responsibility -- > the new type stuff isn't documented enough yet. :-( I think I got everything right as far as the new type code goes (and it works in most cases. For reference, the gtk.Button instance is a new style type (subclassed from other new style types implemented in C, single inheritance). The base class (GObject) implements the tp_compare slot in the type object, and doesn't implement rich comparisons: >>> import gtk >>> gtk.Button.__cmp__ <slot wrapper '__cmp__' of 'GObject' objects> >>> b = gtk.Button() >>> b.__cmp__ <method-wrapper object at 0x8268df8> I don't think I am doing anything weird in the tp_compare slot (although the function would only handle GObject instances). > > > This is too new to have tarballs yet (lots of hairballs perhaps...). > > Everything is coming out of various CVS repositories. If you want to waste > > a half a day getting all the Gtk build tools and libraries correct, I can > > set you up, but I suspect you may have better things to do with your time. > > Yup. Well, the 1.3.9 releases of glib and gtk are usable. The last development release of pygtk (1.99.2) had some issues, as it was released just after I merged the branch where I ported things to 2.2. I will put out a 1.99.3 release soon which should work a bit better. James. -- Email: james@daa.com.au WWW: http://www.daa.com.au/~james/ From skip@pobox.com (Skip Montanaro) Thu Oct 4 14:41:48 2001 From: skip@pobox.com (Skip Montanaro) (Skip Montanaro) Date: Thu, 4 Oct 2001 08:41:48 -0500 Subject: [pygtk] Re: [Python-Dev] a bit confused about the new type/class stuff In-Reply-To: <200110040128.VAA32301@cj20424-a.reston1.va.home.com> References: <15291.25709.109749.875704@beluga.mojam.com> <200110040128.VAA32301@cj20424-a.reston1.va.home.com> Message-ID: <15292.26396.278940.686089@beluga.mojam.com> >> >>> l >> [<Button object (GtkButton) at 0x8293134>, <Button object (GtkButton) at 0x8296ac4>, <Button object (GtkButton) at 0x8397fb4>] >> >>> b = l[0] >> >>> b in l >> Traceback (most recent call last): >> File "<stdin>", line 1, in ? >> TypeError: Button.__cmp__(x,y) requires y to be a 'Button', not a 'instance' A brief conversation with Guido cleared up what was happening. In my class hierarchy I didn't implement __cmp__, but I did implement __getattr__, which was delegating most things to the underlying Gtk widget, so b.__cmp__(l) became b._real_widget.__cmp(l) which then barfed. Writing a __cmp__ method that does the right thing cleared up the problem. Skip From aahz@rahul.net Thu Oct 4 16:02:05 2001 From: aahz@rahul.net (Aahz Maruch) Date: Thu, 4 Oct 2001 08:02:05 -0700 (PDT) Subject: [Python-Dev] Backward compatibility In-Reply-To: <LNBBLJKPBEHFEDALKOLCIELHLOAA.tim.one@home.com> from "Tim Peters" at Oct 01, 2001 02:47:04 PM Message-ID: <20011004150205.9A529E906@waltz.rahul.net> Tim Peters wrote: > > I pay no attention to "backwards compatibility" issues in the libraries we > ship, and have Guido's blessing for that ornery attitude: the libs we ship > should do the best possible job in the context of the release they're > shipped in. It's explicitly not a PythonLabs goal that people be able to > pick modules out of a release and use them in an earlier release (although > it may be a goal of some developers for some modules -- but then the burden > is on them to ensure it works for the module and release combinations they > care about). As one of the frequent ranters about compatibility issues, I thoroughly endorse this approach WRT "batteries included" as long as the APIs stay the same. -- --- Aahz (@pobox.com) Hugs and backrubs -- I break Rule 6 <*> http://www.rahul.net/aahz/ Androgynous poly kinky vanilla queer het Pythonista We must not let the evil of a few trample the freedoms of the many. From aahz@rahul.net Thu Oct 4 16:32:54 2001 From: aahz@rahul.net (Aahz Maruch) Date: Thu, 4 Oct 2001 08:32:54 -0700 (PDT) Subject: [Python-Dev] Pmw broken under 2.2a4? In-Reply-To: <000101c14b8f$db4938a0$3d01a8c0@plstn1.sfba.home.com> from "Paul F. Dubois" at Oct 02, 2001 03:16:16 PM Message-ID: <20011004153254.CD682E906@waltz.rahul.net> Paul F. Dubois wrote: > > Building Pmw 0.8.5 under 2.2a4 I got syntax errors of the 'inconsistent > use of tabs and spaces' kind. I'm not sure who I should tell but I > speculate that I just did. Try http://pmw.sourceforge.net/ -- --- Aahz (@pobox.com) Hugs and backrubs -- I break Rule 6 <*> http://www.rahul.net/aahz/ Androgynous poly kinky vanilla queer het Pythonista We must not let the evil of a few trample the freedoms of the many. From loewis@informatik.hu-berlin.de Thu Oct 4 18:12:51 2001 From: loewis@informatik.hu-berlin.de (Martin von Loewis) Date: Thu, 4 Oct 2001 19:12:51 +0200 (MEST) Subject: [Python-Dev] Making setup.py a little more flexible Message-ID: <200110041712.TAA25396@pandora.informatik.hu-berlin.de> > Whaddya think? Good enough bandaid solution? -1. First of all, a bandaid for the specific problem is already in place: you can configure everything through Modules/Setup, in particular if you find that setup.py doesn't do the right thing. Modules/Setup allow much more detailed changes than what your changes to setup.py offers, plus it is an established solution to this problem. Furthermore, your specific patch seems incomplete: You have to run setup.py manually, which means that you cannot use a plain "make". There should be atleast minimal interfacing to Makefile.pre.in for this patch to be acceptable. Regards, Martin From walter@livinglogic.de Thu Oct 4 19:22:01 2001 From: walter@livinglogic.de (Walter =?ISO-8859-1?Q?D=F6rwald?=) Date: Thu, 04 Oct 2001 20:22:01 +0200 Subject: [Python-Dev] Proposed PEP: Codec error handling callbacks Message-ID: <3BBCA8C9.7050901@livinglogic.de> Here's a first try at a PEP for making Python's codec error handling customizable. This has been briefly discussed before on the I18N-SIG mailing list. A sample implementation of the proposed feature is available as a SourceForge patch. Bye, Walter D=F6rwald -------------------------------------------------------------------------= -- PEP: ??? Title: Codec error handling callbacks Version: $Revision$ Last-Modified: $Date$ Author: walter@livinglogic.de (Walter D=F6rwald) Status: Draft Type: Standards Track Python-Version: 2.3 Created: ??-???-2001 Post-History: 04-Oct-2001 Abstract This PEP aims at extending Python's fixed codec error handling schemes with a more flexible callback based approach. Python currently uses a fixed error handling for codec error handlers. This PEP describes a mechanism which allows Python to use function callbacks as error handlers. With these more flexible error handlers it is possible to add new functionality to existing codecs by e.g. providing fallback solutions or different encodings for cases where the standard codec mapping does not apply. Problem A typical case is where a character encoding does not support the full range of Unicode characters. For these cases many high level protocols support a way of escaping a Unicode character (e.g. Python itself support the \x, \u and \U convention, XML supports character references via &#xxxx; etc.). When implementing such an encoding algorithm, a problem with the current implementation of the encode method of Unicode objects becomes apparent: For determining which characters are unencodable by a certain encoding, every single character has to be tried, because encode does not provide any information about the location of the error(s), so # (1) us =3D u"xxx" s =3D us.encode(encoding) has to be replaced by # (2) us =3D u"xxx" v =3D [] for c in us: try: v.append(c.encode(encoding)) except UnicodeError: v.append("&#" + ord(c) + ";") s =3D "".join(v) This slows down encoding dramatically as now the loop through the string is done in Python code and no longer in C code. Furthermore this solution poses problems with stateful encodings. For example UTF16 uses a Byte Order Mark at the start of the encoded byte string to specify the byte order. Using (2) with UTF16, results in an 8 bit string with a BOM between every character. To work around this problem, a stream writer - which keeps state between calls to the encoding function - has to be used: # (3) us =3D u"xxx" import codecs, cStringIO as StringIO writer =3D codecs.lookup(encoding)[3] v =3D StringIO.StringIO() uv =3D writer(v) for c in us: try: uv.write(c) except UnicodeError: uv.write(u"&#%d;" % ord(c)) s =3D str(v) To compare the speed of (1) and (3) the following test script has been used: # (4) import time us =3D u"=E4a"*1000000 encoding =3D "ascii" import codecs, cStringIO as StringIO t1 =3D time.time() s1 =3D us.encode(encoding, "replace") t2 =3D time.time() writer =3D codecs.lookup(encoding)[3] v =3D StringIO.StringIO() uv =3D writer(v) for c in us: try: uv.write(c) except UnicodeError: uv.write(u"?") s2 =3D v.getvalue() t3 =3D time.time() assert(s1=3D=3Ds2) print "1:", t2-t1 print "2:", t3-t2 print "factor:", (t3-t2)/(t2-t1) On Linux with Python 2.1 this gives the following output: 1: 0.395456075668 2: 126.909595966 factor: 320.919575586 i.e. (3) is 320 times slower than (1). Solution 1 Add a position attribute to UnicodeError instances. When the encoder encounters an unencodable character it raises an exception that specifies the position in the Unicode object where the unencodable character occurs. The application can then reencode the Unicode object up to the offending character, replace the offending character with something appropriate and retry encoding with the rest of the Unicode string until encoding is finished. A stream writer will write everything up to the offending character and then raise an exception, so the application only has to replace the offending character and retry the rest of the string. Advantage Requires minor changes to all the encoders/stream writers. Disadvantage As the encoder has to be called multiple times, this won't work with stateful encoding, so a stream writer has to be used in all cases. If unencodable characters occur very often Solution 1 will probably not be much faster than (3). E.g. for the string u"a"*10000+u"=E4" all the characters but the last one will have to be encoded twice when using an encoder (but only once when using a stream writer). This solution is specific to encoding and can't be extended to decoding easily. Solution 2 Add additional error handling names for every needed replacement scheme (e.g. "xmlcharrefreplace" for "&#%d;" or "escapereplace" for "\\x%02x" / "\\u%04x" / "\\U%08x") Advantage Requires minor changes to all encoders/stream writers. As the replacement scheme is implemented directly in the encoder this should be the fastest solution. Disadvantages The available replacement schemes are hardwired. Additional replacement schemes require that all encoders/decoders are changed again. This is especially problematic for decoding where users might want to implement various error handlers for handling broken text files. Solution 3 The errors argument is no longer a string, but a callback function: This callback function gets passed the original unicode object and the position of the unencodable character and returns a new unicode object that should be encoded instead of the unencodable one. (Note that this requires that the encoder *must* be able to encode what is returned from the handler. If not a normal UnicodeError will be raised.) Example code could look like this: us =3D u"xxx" def xmlescape(uni, pos): return u"&#%d;" % ord(uni[pos]) s =3D us.encode(encode, xmlescape) Advantages This makes the error handling completely customizable. The error handler may choose to raise an exception or do any kind of replacement required. The interface between the encoder and the error handler can be designed in a way that this mechanism can be used for decoding too: The original 8bit string is passed to the error handler and the error handler returns a replacement unicode object and a resyncronization position where decoding should continue. Disadvantages This solutions requires changes to every C function that has "const char *errors" arguments. (e.g. PyUnicode_EncodeLatin1( const Py_UNICODE *p, int size, const char *errors) has to be changed to PyUnicode_EncodeLatin1( const Py_UNICODE *p, int size, PyObject *errors) (To provide backwards compatibility the PyUnicode_EncodeLatin1 signature remains the same, a new function PyUnicode_EncodeLatin1Ex is introduced with the new signature. PyUnicode_EncodeLatin1 simply calls the new function.) Solution 4 The errors argument is still a string, but this string is used to lookup a error handling callback function from a callback registry. Advantage No changes to the Python C API are required. Well known error handling schemes could be implemented directly in the encoder/decoder for maximum speed. Like solution 3 this can be done for encoders and decoders. Disadvantages Currently all encoding/decoding functions have arguments const Py_UNICODE *p, int size or const char *p, int size to specify the unicode characters/8bit characters to be encoded/decoded. In case of a error a new unicode or str object has to be created from this information and passed to the error handler. This has to be done either for every occuring error or once on the first error and the result object must be kept until the end of the function in case another error occurs. Most functions that call the codec functions work with unicode/str objects anyway, so they have to extract the const Py_UNICODE */int arguments and pass it to the functions, which has to reconstruct the object in case of an error. Sample implementation A sample implementation is available on SourceForge [1]. This patch implements a combination of solutions 3 and 4, i.e. it is possible to pass functions *and* registered callback names to unicode.encode. The interface between the encoder and the handler has been extended to be able to support the same interface for encoding and decoding. The encoder/decoder passes a tuple to the callback with the following entries: 0: the name of the encoding 1: the original Unicode object/the original 8bit string 2: the position of the unencodable character/byte 3: the reason why the character/byte can't be encoded/decoded as a string (e.g. "character not in range(128)" for encoding or "unexpected end of data", "invalid code byte" etc. for decoding) 4: an additional object that can be used to pass state information to the callback. All implemented encoders/decoders currently pass None for this. The callback must return a tuple with the following info: 0: a Unicode object which will be encoded instead of the offending character (for encoders). A Unicode object that will be used as a replacement for the undecodable bytes in the decoded Unicode object (for decoders) 1: a position where the encoding/decoding process should continue after processing the replacement string. The patch includes several preregistered encode error handlers schemes with the following names: "strict", "ignore", "replace", "xmlcharrefreplace", "escapereplace" and decode error handlers with the names: "strict", "ignore", "replace" The patch includes the other change to the C API described in Solution 4. The parameters const Py_UNICODE *p, int size have been replace by PyObject *p so that all functions get the Unicode object directly as a PyObject * and pass this directly along to the error callback. For further details see the patch on SourceForge. References [1] Python patch #432401 "unicode encoding error callbacks" =20 http://sourceforge.net/tracker/?group_id=3D5470&atid=3D305470&func=3Ddeta= il&aid=3D432401 [2] Previous discussions on this topics in the I18N-SIG mailing list http://mail.python.org/pipermail/i18n-sig/2000-December/000587.h= tml http://mail.python.org/pipermail/i18n-sig/2001-June/001173.html Copyright This document has been placed in the public domain. =0C Local Variables: mode: indented-text indent-tabs-mode: nil End: From skip@pobox.com (Skip Montanaro) Fri Oct 5 03:12:21 2001 From: skip@pobox.com (Skip Montanaro) (Skip Montanaro) Date: Thu, 4 Oct 2001 21:12:21 -0500 Subject: [Python-Dev] Backward compatibility Message-ID: <15293.5893.817970.754724@beluga.mojam.com> Aahz> As one of the frequent ranters about compatibility issues, I Aahz> thoroughly endorse this approach WRT "batteries included" as long Aahz> as the APIs stay the same. I don't think APIs have much to do with this particular item. No API changed in xmlrpclib.py, but the code became incompatible with pre-2.0 releases. I checked in a change to xmlrpclib.py that used "import foo as bar". The question I raised implicitly by making an incompatible change to xmlrpclib.py was "is it okay to use all the features of the current release in a core module?". Tim's answer was a qualified "yes", the exception being for modules maintained by other authors. In PythonLabs view it's ultimately Fredrik's responsibility to maintain backward compatibility if he so chooses. I realized it about 30 seconds after I checked in the change, which was why I mentioned it. I didn't know if he maintains a separate copy for backwards compatible releases or relies on the code now in the Python CVS repository. In retrospect, I should probably have sent him a note instead of checking in the change. Skip From ask@valueclick.com Fri Oct 5 09:40:28 2001 From: ask@valueclick.com (Ask Bjoern Hansen) Date: Fri, 5 Oct 2001 01:40:28 -0700 (PDT) Subject: [Python-Dev] Crude Python->Parrot compiler In-Reply-To: <200109261751.f8QHppn23941@odiug.digicool.com> Message-ID: <Pine.LNX.4.33.0110050138460.4496-100000@impatience.valueclick.com> On Wed, 26 Sep 2001, Guido van Rossum wrote: > > Question for Guido and Simon: Should I check this into sandbox/ in the > > Python CVS tree, or do you want to pull it into the Parrot CVS? I'd > > like to have it in a public CVS tree someplace, preferably one where I > > have checkin privileges. sandbox/ is fine with me. > > If Simon can give you write access in a corner of the Parrot tree, > that might be the best solution; if not, sandbox is fine with me too. I didn't see Simon follow up on this. I would think that we can get Andrew an account. Andrew, sign up for an account at http://dev.perl.org/auth/account, then if Simon will just say the word I can flip the bits to give you commit access to parrot/languages/python/ or something like that. :-) - ask -- ask bjoern hansen, http://ask.netcetera.dk/ !try; do(); more than a billion impressions per week, http://valueclick.com From thomas.heller@ion-tof.com Fri Oct 5 10:58:12 2001 From: thomas.heller@ion-tof.com (Thomas Heller) Date: Fri, 5 Oct 2001 11:58:12 +0200 Subject: [Python-Dev] imputil and modulefinder replacements References: <3BB204E3.23339.3A9F068C@localhost> Message-ID: <064a01c14d84$3ed0fcf0$e000a8c0@thomasnotebook> From: "Gordon McMillan" <gmcm@hypernet.com> > Dev-er's > > After numerous false-starts and dead-ends, I've come up with > two modules that do what imputil and modulefinder do, but > better. > > Code and detailed descriptions are available here: > > http://www.mcmillan-inc.com/importhooks.html > Gordon, I've read your descriptions and IMO this is a great design for a new import-utility and a new module-finder. Some comments: To fully simulate python's behaviour for import, the case of the filename must be checked on windows. I found only two possibilities to do this - the first one is to use os.listdir() which is probably expensive, but it returns filenames with the actual case. Second would be to use Mark's win32api.FindFiles(), but this may not be available. Your metapath models Python's import policy. One part of the policy is that already loaded modules are fetched from sys.modules instead of imported again. Should this behaviour also be modeled by a SysModulesCacheDirector on your metapath? What about reload? Shouldn't a new import-util module also implement a reload-replacement? > I would like to propose these (or something quite like them) as > replacements for the official versions. The code is quite similar > (in fact, the modulefinder code could have been written by > subclassing the imputil stuff, but I wrote them the other way > 'round). > It seems noone cares about this. imputil is in the distribution, but is it really 'official'? > If the charter of the Import-SIG is not as dead as the list is, I > would promote the basic structure as a potential model for a > reformed import. > > For now, though, it's enough to consider the code. The > differences are too extreme to consider these patches, but the > subject hardly seems PEPable so I bring it up here. > > > - Gordon > Thomas From simon@netthink.co.uk Fri Oct 5 12:12:14 2001 From: simon@netthink.co.uk (Simon Cozens) Date: Fri, 5 Oct 2001 12:12:14 +0100 Subject: [Python-Dev] Crude Python->Parrot compiler In-Reply-To: <Pine.LNX.4.33.0110050138460.4496-100000@impatience.valueclick.com> References: <200109261751.f8QHppn23941@odiug.digicool.com> <Pine.LNX.4.33.0110050138460.4496-100000@impatience.valueclick.com> Message-ID: <20011005121214.B743@netthink.co.uk> On Fri, Oct 05, 2001 at 01:40:28AM -0700, Ask Bjoern Hansen wrote: > I didn't see Simon follow up on this. I would think that we can get > Andrew an account. I think we decided off-list that Guido would be OK with it going in Python CVS for now. Also, www.parrotcode.org -- "He was a modest, good-humored boy. It was Oxford that made him insufferable." From gmcm@hypernet.com Fri Oct 5 12:58:05 2001 From: gmcm@hypernet.com (Gordon McMillan) Date: Fri, 5 Oct 2001 07:58:05 -0400 Subject: [Python-Dev] imputil and modulefinder replacements In-Reply-To: <064a01c14d84$3ed0fcf0$e000a8c0@thomasnotebook> Message-ID: <3BBD680D.8967.671A64B6@localhost> Thomas, > Some comments: > To fully simulate python's behaviour for import, the case of the > filename must be checked on windows. True. Sigh. > I found only two > possibilities to do this - the first one is to use os.listdir() > which is probably expensive, but it returns filenames with the > actual case. Second would be to use Mark's win32api.FindFiles(), > but this may not be available. > > Your metapath models Python's import policy. One part of the > policy is that already loaded modules are fetched from > sys.modules instead of imported again. Should this behaviour also > be modeled by a SysModulesCacheDirector on your metapath? That's already done by the ImportManager as soon as he's got a candidate fqname. > What about reload? Shouldn't a new import-util module also > implement a reload-replacement? Yes, it should. > > I would like to propose these (or something quite like them) as > > replacements for the official versions. The code is quite > > similar (in fact, the modulefinder code could have been written > > by subclassing the imputil stuff, but I wrote them the other > > way 'round). > > > It seems noone cares about this. imputil is in the distribution, > but is it really 'official'? The entire subject seems to have dropped off radar, after starting out as highly controversial (the Import-SIG was started so the ihooks-partisans could hash it out with the imputil- partisans). Import hacks are more common than ever, but they're all home-grown now. - Gordon From thomas.heller@ion-tof.com Fri Oct 5 13:39:33 2001 From: thomas.heller@ion-tof.com (Thomas Heller) Date: Fri, 5 Oct 2001 14:39:33 +0200 Subject: [Python-Dev] imputil and modulefinder replacements References: <3BBD680D.8967.671A64B6@localhost> Message-ID: <067f01c14d9a$ca0f30a0$e000a8c0@thomasnotebook> [Thomas] > > Your metapath models Python's import policy. One part of the > > policy is that already loaded modules are fetched from > > sys.modules instead of imported again. Should this behaviour also > > be modeled by a SysModulesCacheDirector on your metapath? > [Gordon] > That's already done by the ImportManager as soon as he's got > a candidate fqname. I know this. I thought of moving this from ImportManager into an ImportDirector. Probably doesn't matter too much. > The entire subject seems to have dropped off radar, after > starting out as highly controversial (the Import-SIG was started > so the ihooks-partisans could hash it out with the imputil- > partisans). Import hacks are more common than ever, but > they're all home-grown now. You thought of replacing the import hacks by custom import policies implemented with imputil or iu4? No chance, IMO. Several reasons I can think of: - Import hacks have this cool touch... - Import hacks start out to help the developers, but they never get cleaned up for the poor user (pmw being an exception!) Thomas From guido@python.org Fri Oct 5 14:06:32 2001 From: guido@python.org (Guido van Rossum) Date: Fri, 05 Oct 2001 09:06:32 -0400 Subject: [Python-Dev] imputil and modulefinder replacements In-Reply-To: Your message of "Fri, 05 Oct 2001 11:58:12 +0200." <064a01c14d84$3ed0fcf0$e000a8c0@thomasnotebook> References: <3BB204E3.23339.3A9F068C@localhost> <064a01c14d84$3ed0fcf0$e000a8c0@thomasnotebook> Message-ID: <200110051306.JAA24439@cj20424-a.reston1.va.home.com> > It seems noone cares about this. imputil is in the distribution, > but is it really 'official'? I want to care, but I'm already spread too thin. I don't find 'imputil' particularly official -- but Greg Stein pushed hard to get it in the distribution. --Guido van Rossum (home page: http://www.python.org/~guido/) From gmcm@hypernet.com Fri Oct 5 14:25:39 2001 From: gmcm@hypernet.com (Gordon McMillan) Date: Fri, 5 Oct 2001 09:25:39 -0400 Subject: [Python-Dev] imputil and modulefinder replacements In-Reply-To: <067f01c14d9a$ca0f30a0$e000a8c0@thomasnotebook> Message-ID: <3BBD7C93.29505.676A8D8C@localhost> [Gordon] > > The entire subject seems to have dropped off radar, after > > starting out as highly controversial (the Import-SIG was > > started so the ihooks-partisans could hash it out with the > > imputil- partisans). Import hacks are more common than ever, > > but they're all home-grown now. [Thomas] > You thought of replacing the import hacks by custom import > policies implemented with imputil or iu4? No chance, IMO. That's what ihooks was written for, and nobody ever used it, either. > Several reasons I can think of: > - Import hacks have this cool touch... Ironically, they usually are justified as "making it easier on the user" (where they really mean programmer). Which is crap, because when the programmer has trouble with X.A.fribble(), he goes looking for the source. But there is no X/A.py. > - Import hacks start out to help the developers, but they never > get cleaned up for the poor user (pmw being an exception!) And they help constrain Python to being a developer-only language. The existing import mechanism is poorly understood (just see how many wrong answers an import question on c.l.py gets) and home to some major warts (such as the fact that relative and absolute imports are spelled the same way). Greg wrote imputil as a first step towards fixing those kinds of problems. under-the-spreading-entropy-ly y'rs - Gordon From thomas.heller@ion-tof.com Fri Oct 5 14:36:25 2001 From: thomas.heller@ion-tof.com (Thomas Heller) Date: Fri, 5 Oct 2001 15:36:25 +0200 Subject: [Python-Dev] imputil and modulefinder replacements References: <3BBD7C93.29505.676A8D8C@localhost> Message-ID: <073001c14da2$baa54ac0$e000a8c0@thomasnotebook> > The existing import mechanism is poorly understood (just see > how many wrong answers an import question on c.l.py gets) ... Note that this question has recently been asked by Michael Abbot on c.l.p, for now there are no answers: Is there up to date documentation for package support in Python 2? Section 6.11 of the "Python Reference Manual" has the following nice quote: [XXX Can't be bothered to spell this out right now; see the URL http://www.python.org/doc/essays/packages.html for more details, also about how the module search works from inside a package.] and the referred URL documents Python 1.5. I presume that nothing significant has changed recently, but it's certainly disconcerting for something as fundamental as module importing to not actually be part of the core language documentation! Thomas From guido@python.org Fri Oct 5 15:36:25 2001 From: guido@python.org (Guido van Rossum) Date: Fri, 05 Oct 2001 10:36:25 -0400 Subject: [Python-Dev] Need a way to test for 8-bit-or-unicode-string Message-ID: <200110051436.KAA24983@cj20424-a.reston1.va.home.com> I'm finding more and more the need to test whether an object is a string, including both 8-bit and Unicode strings. Current practice seems to be: if type(x) in (str, unicode): or (more verbose but more b/w compatible): if type(x) in (types.StringType, types.UnicodeType): or (using a variable recently added to types -- I'm not sure this was a good idea): if type(x) in types.StringTypes: all of which break if type(x) is a *subclass* of str or unicode. The alternative: if isinstance(x, str) or isinstance(x, unicode): is apparently too much typing. Some alternatives that have been proposed already: - Create a common base class of str and unicode, which should be an abstract class. This is the most OO solution, but I can't think of a good name; abstractstring is too long, AbstractString or String are uncommon naming conventions for built-in types, 'string' would almost work except that it's already the name of a very common module.[*] - Make str a subclass of unicode (or vice versa). This can't be done because subclassing requires implementation inheritance, in particular the instance structure layout must overlap. Also, this would make it hard to check for either str or unicode. - Create a new service function, IsString(x) or isString(x) or isstring(x), that's a shortcut for "isinstance(x, str) or isinstance(x, unicode)". The question them becomes where to put this: as a builtin, in types.py, or somewhere else... Preferences please? --Guido van Rossum (home page: http://www.python.org/~guido/) [*] For a while I toyed with the idea of calling the abstract base class 'string', and hacking import so that sys.modules['string'] is the string class. The abstract base class should then have methods that invoke the concrete implementations, so that string.split(s) would be the same as s.split(). This would be compatible with previous uses of the string module! string.letters etc. could then be class variables. Unfortunately this broke down when I realized that the signature of string.join() is wrong for the string module: the the string.join function is string.join(sequence, stringobject) while the signature of the string method is join(stringobject, sequence). So much for that idea... :-) (BTW this shows to me again that the method signature is right and the function signature is wrong. But even my time machine isn't powerful enough to fix this.) (Hm, it could be saved by making string.join() accept the arguments in either order. Gross. :-) From nas@python.ca Fri Oct 5 16:11:36 2001 From: nas@python.ca (Neil Schemenauer) Date: Fri, 5 Oct 2001 08:11:36 -0700 Subject: [Python-Dev] Need a way to test for 8-bit-or-unicode-string In-Reply-To: <200110051436.KAA24983@cj20424-a.reston1.va.home.com>; from guido@python.org on Fri, Oct 05, 2001 at 10:36:25AM -0400 References: <200110051436.KAA24983@cj20424-a.reston1.va.home.com> Message-ID: <20011005081136.A25134@glacier.arctrix.com> Guido van Rossum wrote: > > if isinstance(x, str) or isinstance(x, unicode): > > is apparently too much typing. Perhaps we could extend isinstance(). How about isinstance(x, str, unicode) or isinstance(x, (str, unicode)) This is a common problem not limited to string types. I often want to test if something is a tuple or a list for example. Neil From guido@python.org Fri Oct 5 16:20:30 2001 From: guido@python.org (Guido van Rossum) Date: Fri, 05 Oct 2001 11:20:30 -0400 Subject: [Python-Dev] Need a way to test for 8-bit-or-unicode-string In-Reply-To: Your message of "Fri, 05 Oct 2001 08:11:36 PDT." <20011005081136.A25134@glacier.arctrix.com> References: <200110051436.KAA24983@cj20424-a.reston1.va.home.com> <20011005081136.A25134@glacier.arctrix.com> Message-ID: <200110051520.LAA25303@cj20424-a.reston1.va.home.com> > Guido van Rossum wrote: > > > > if isinstance(x, str) or isinstance(x, unicode): > > > > is apparently too much typing. > > Perhaps we could extend isinstance(). How about > > isinstance(x, str, unicode) > > or > > isinstance(x, (str, unicode)) > > This is a common problem not limited to string types. I often want to > test if something is a tuple or a list for example. You may be on to something. It certainly saves me inventing a new name! I'll think about it a bit more... --Guido van Rossum (home page: http://www.python.org/~guido/) From barry@zope.com Fri Oct 5 16:27:01 2001 From: barry@zope.com (Barry A. Warsaw) Date: Fri, 5 Oct 2001 11:27:01 -0400 Subject: [Python-Dev] Need a way to test for 8-bit-or-unicode-string References: <200110051436.KAA24983@cj20424-a.reston1.va.home.com> <20011005081136.A25134@glacier.arctrix.com> Message-ID: <15293.53573.251652.664773@anthem.wooz.org> GvR> if isinstance(x, str) or isinstance(x, unicode): > > is GvR> apparently too much typing. NS> Perhaps we could extend isinstance(). How about NS> isinstance(x, str, unicode) NS> or NS> isinstance(x, (str, unicode)) NS> This is a common problem not limited to string types. I often NS> want to test if something is a tuple or a list for example. That's an interesting idea, but please use the former signature, not the latter. And what would it return? It needs to return a true value on success, but maybe instead of returning 1, it might be more useful to return the type argument that matched, e.g.: >>> isinstance('', str, unicode) <type 'str'> >>> isinstance(u'', str, unicode) <type 'unicode'> >>> isinstance((), list, dictionary, tuple) <type 'tuple'> >>> isinstance(7, list, dictionary, tuple) 0 -Barry From fdrake@acm.org Fri Oct 5 16:21:50 2001 From: fdrake@acm.org (Fred L. Drake, Jr.) Date: Fri, 5 Oct 2001 11:21:50 -0400 Subject: [Python-Dev] Need a way to test for 8-bit-or-unicode-string In-Reply-To: <200110051520.LAA25303@cj20424-a.reston1.va.home.com> References: <200110051436.KAA24983@cj20424-a.reston1.va.home.com> <20011005081136.A25134@glacier.arctrix.com> <200110051520.LAA25303@cj20424-a.reston1.va.home.com> Message-ID: <15293.53262.479355.538873@grendel.zope.com> Neil Schemenauer wrote: > isinstance(x, str, unicode) ... > This is a common problem not limited to string types. I often want to > test if something is a tuple or a list for example. Guido van Rossum writes: > You may be on to something. It certainly saves me inventing a new > name! I'll think about it a bit more... I like this approach. It also doesn't preclude the other changes. -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> PythonLabs at Zope Corporation From jim@interet.com Fri Oct 5 16:31:32 2001 From: jim@interet.com (James C. Ahlstrom) Date: Fri, 05 Oct 2001 11:31:32 -0400 Subject: [Python-Dev] imputil and modulefinder replacements References: <3BB204E3.23339.3A9F068C@localhost> <064a01c14d84$3ed0fcf0$e000a8c0@thomasnotebook> Message-ID: <3BBDD254.32DC4071@interet.com> Thomas Heller wrote: > > From: "Gordon McMillan" <gmcm@hypernet.com> > > I would like to propose these (or something quite like them) as > > replacements for the official versions. The code is quite similar > > (in fact, the modulefinder code could have been written by > > subclassing the imputil stuff, but I wrote them the other way > > 'round). > > > It seems noone cares about this. imputil is in the distribution, > but is it really 'official'? I care about it. I am using imputil to implement "jar" files, zip files of Python libraries. Greg worked hard to get imputil into the distribution, and I appreciate his effort. But I always wondered if imputil was 'official'. Users on c.l.p frequently want to ship a stand-alone Python. The current answer to this is freeze, but this requires a C compiler and some work. A standard jar file implementation would satisfy this need. What I am thinking of is documentation which says something like: "Zip up the Python libraries into pylibs.zip and put that file in the same directory as python[.exe] and everything Just Works." My interest in imputil and ImportManager is to implement jar files, and neither does this well. The problem is always how to get the SpecialImporter.py imported first, and how to hack the imports that the SpecialImporter.py may need to import. In iu4.py we see the use of posix.stat to replace the os module, and the import of only built-in modules sys, imp, and marshal. The same is true of imputil.py. I admit I don't really understand package imports. But I feel they are an unsolved problem, because package authors always seem to write custom ones. A first-class solution would have these features IMHO: 1) Access to libraries in zip files is implemented in C (I volunteer). 2) All Python library modules can be imported by C code, either from directories or zip files. 3) Package import is not implemented in C, it is implemented by a Python module which is part of the standard library. 4) If an import of A.B is attempted, and a function A.MyCustomImporter() exists, then that is used instead of the default. 5) More or better ideas from you all. Jim Ahlstrom From fdrake@acm.org Fri Oct 5 16:24:22 2001 From: fdrake@acm.org (Fred L. Drake, Jr.) Date: Fri, 5 Oct 2001 11:24:22 -0400 Subject: [Python-Dev] Need a way to test for 8-bit-or-unicode-string In-Reply-To: <15293.53573.251652.664773@anthem.wooz.org> References: <200110051436.KAA24983@cj20424-a.reston1.va.home.com> <20011005081136.A25134@glacier.arctrix.com> <15293.53573.251652.664773@anthem.wooz.org> Message-ID: <15293.53414.472587.851330@grendel.zope.com> Barry A. Warsaw writes: > And what would it return? It needs to return a true value on success, > but maybe instead of returning 1, it might be more useful to return > the type argument that matched, e.g.: Ugh! If that's what you want, make it explicit in the code: for t in ListType, TupleType, ...: if isinstance(obj, t): break else: raise Exception('no match!') -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> PythonLabs at Zope Corporation From guido@python.org Fri Oct 5 16:43:44 2001 From: guido@python.org (Guido van Rossum) Date: Fri, 05 Oct 2001 11:43:44 -0400 Subject: [Python-Dev] Need a way to test for 8-bit-or-unicode-string In-Reply-To: Your message of "Fri, 05 Oct 2001 11:27:01 EDT." <15293.53573.251652.664773@anthem.wooz.org> References: <200110051436.KAA24983@cj20424-a.reston1.va.home.com> <20011005081136.A25134@glacier.arctrix.com> <15293.53573.251652.664773@anthem.wooz.org> Message-ID: <200110051543.LAA25454@cj20424-a.reston1.va.home.com> > GvR> if isinstance(x, str) or isinstance(x, unicode): > > is > GvR> apparently too much typing. > > NS> Perhaps we could extend isinstance(). How about > > NS> isinstance(x, str, unicode) > > NS> or > > NS> isinstance(x, (str, unicode)) > > NS> This is a common problem not limited to string types. I often > NS> want to test if something is a tuple or a list for example. > > That's an interesting idea, but please use the former signature, not > the latter. Why? The varargs signature makes it less convenient (and less efficient!) to pre-calculate the list of types. It also makes it harder to implement this in PyObject_IsInstance(), so that C code can use this convenience. Finally, if I already know the meaning of isinstance(x, y), then when I encounter isinstance(x, (y, z)) for the first time, it's very easy to guess the meaning. The meaning of isinstance(x, y, z) is much more murky: z could be an optional argument specifying some other modification of the basic isinstance(). > And what would it return? It needs to return a true value on success, > but maybe instead of returning 1, it might be more useful to return > the type argument that matched, e.g.: > > >>> isinstance('', str, unicode) > <type 'str'> > >>> isinstance(u'', str, unicode) > <type 'unicode'> > >>> isinstance((), list, dictionary, tuple) > <type 'tuple'> > >>> isinstance(7, list, dictionary, tuple) > 0 Here I agree with Fred: too much hackery combined in one function. And what's the use case? --Guido van Rossum (home page: http://www.python.org/~guido/) From jeremy@zope.com Fri Oct 5 16:47:11 2001 From: jeremy@zope.com (Jeremy Hylton) Date: Fri, 5 Oct 2001 11:47:11 -0400 (EDT) Subject: [Python-Dev] Need a way to test for 8-bit-or-unicode-string In-Reply-To: <15293.53414.472587.851330@grendel.zope.com> References: <200110051436.KAA24983@cj20424-a.reston1.va.home.com> <20011005081136.A25134@glacier.arctrix.com> <15293.53573.251652.664773@anthem.wooz.org> <15293.53414.472587.851330@grendel.zope.com> Message-ID: <15293.54783.743905.190860@slothrop.digicool.com> >>>>> "FLD" == Fred L Drake, <fdrake@acm.org> writes: FLD> for t in ListType, TupleType, ...: FLD> if isinstance(obj, t): FLD> break FLD> else: FLD> raise Exception('no match!') Or: try: raise obj except (ListType, TupleType): pass except: raise TypeError, ... with-apologies-to-tim-ly y'rs, Jeremy From guido@python.org Fri Oct 5 16:46:57 2001 From: guido@python.org (Guido van Rossum) Date: Fri, 05 Oct 2001 11:46:57 -0400 Subject: [Python-Dev] Need a way to test for 8-bit-or-unicode-string In-Reply-To: Your message of "Fri, 05 Oct 2001 11:27:01 EDT." <15293.53573.251652.664773@anthem.wooz.org> References: <200110051436.KAA24983@cj20424-a.reston1.va.home.com> <20011005081136.A25134@glacier.arctrix.com> <15293.53573.251652.664773@anthem.wooz.org> Message-ID: <200110051546.LAA25549@cj20424-a.reston1.va.home.com> Here's an implementation of Neil's idea. *** abstract.c 2001/10/01 17:10:18 2.83 --- abstract.c 2001/10/05 15:49:04 *************** *** 1805,1810 **** --- 1805,1829 ---- else if (PyType_Check(cls)) { retval = PyObject_TypeCheck(inst, (PyTypeObject *)cls); } + else if (PySequence_Check(cls)) { + int i, n; + + cls = PySequence_Fast( + cls, "isinstance() arg 2 is an unacceptable sequence"); + if (cls == NULL) + return -1; + n = PySequence_Size(cls); + if (n < 0) + retval = -1; + for (i = 0; i < n; i++) { + retval = PyObject_IsInstance( + inst, PySequence_Fast_GET_ITEM(cls, i)); + if (retval != 0) + break; + } + Py_DECREF(cls); + return retval; + } else if (!PyInstance_Check(inst)) { if (__class__ == NULL) { __class__ = PyString_FromString("__class__"); *************** *** 1827,1833 **** if (retval < 0) { PyErr_SetString(PyExc_TypeError, ! "isinstance() arg 2 must be a class or type"); } return retval; } --- 1846,1853 ---- if (retval < 0) { PyErr_SetString(PyExc_TypeError, ! "isinstance() arg 2 must be a class or type " ! "or tuple of those"); } return retval; } --Guido van Rossum (home page: http://www.python.org/~guido/) From fdrake@acm.org Fri Oct 5 16:46:48 2001 From: fdrake@acm.org (Fred L. Drake, Jr.) Date: Fri, 5 Oct 2001 11:46:48 -0400 Subject: [Python-Dev] Need a way to test for 8-bit-or-unicode-string In-Reply-To: <200110051546.LAA25549@cj20424-a.reston1.va.home.com> References: <200110051436.KAA24983@cj20424-a.reston1.va.home.com> <20011005081136.A25134@glacier.arctrix.com> <15293.53573.251652.664773@anthem.wooz.org> <200110051546.LAA25549@cj20424-a.reston1.va.home.com> Message-ID: <15293.54760.113875.151084@grendel.zope.com> Guido van Rossum writes: > Here's an implementation of Neil's idea. Commit this and I'll take care of the documentation. ;-) -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> PythonLabs at Zope Corporation From mal@lemburg.com Fri Oct 5 16:54:05 2001 From: mal@lemburg.com (M.-A. Lemburg) Date: Fri, 05 Oct 2001 17:54:05 +0200 Subject: [Python-Dev] Need a way to test for 8-bit-or-unicode-string References: <200110051436.KAA24983@cj20424-a.reston1.va.home.com> Message-ID: <3BBDD79D.C26D1646@lemburg.com> Guido van Rossum wrote: > > I'm finding more and more the need to test whether an object is a > string, including both 8-bit and Unicode strings. Current practice > seems to be: > > if type(x) in (str, unicode): > > or (more verbose but more b/w compatible): > > if type(x) in (types.StringType, types.UnicodeType): > > or (using a variable recently added to types -- I'm not sure this was > a good idea): > > if type(x) in types.StringTypes: > > all of which break if type(x) is a *subclass* of str or unicode. The > alternative: > > if isinstance(x, str) or isinstance(x, unicode): > > is apparently too much typing. > > Some alternatives that have been proposed already: > > - Create a common base class of str and unicode, which should be an > abstract class. This is the most OO solution, but I can't think of > a good name; abstractstring is too long, AbstractString or String > are uncommon naming conventions for built-in types, 'string' would > almost work except that it's already the name of a very common > module.[*] +1. This would be nice and the same could be done for sequences, file-like objects and other common currently interface-defined object categories. About the naming: how about numberclass, stringclass, sequenceclass, fileclass ?! Then you could write: if isinstance(obj, stringclass): ... which looks OK and is not too much typing. The advantage of this approach is that it can be extended to other types and classes as well (much like you can currently do with the Python exceptions). With the new type logic in place, how hard would it be making the existing built-in types subclasses of these base types ? (also: is there a run-time penalty for this ?) > - Make str a subclass of unicode (or vice versa). This can't be done > because subclassing requires implementation inheritance, in > particular the instance structure layout must overlap. Also, this > would make it hard to check for either str or unicode. -0. This would be hard to get right because the two objects use a very different struct layout. Could be an option in the long run though. > - Create a new service function, IsString(x) or isString(x) or > isstring(x), that's a shortcut for "isinstance(x, str) or > isinstance(x, unicode)". The question them becomes where to put > this: as a builtin, in types.py, or somewhere else... -1. This mechanism can not be extended by e.g. UserStrings. > Preferences please? > > --Guido van Rossum (home page: http://www.python.org/~guido/) > > [*] For a while I toyed with the idea of calling the abstract base > class 'string', and hacking import so that sys.modules['string'] is > the string class. The abstract base class should then have methods > that invoke the concrete implementations, so that string.split(s) > would be the same as s.split(). This would be compatible with > previous uses of the string module! string.letters etc. could then be > class variables. Unfortunately this broke down when I realized that > the signature of string.join() is wrong for the string module: the the > string.join function is string.join(sequence, stringobject) while the > signature of the string method is join(stringobject, sequence). So > much for that idea... :-) (BTW this shows to me again that the method > signature is right and the function signature is wrong. But even my > time machine isn't powerful enough to fix this.) (Hm, it could be > saved by making string.join() accept the arguments in either order. > Gross. :-) Indeed. :-) -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Consulting & Company: http://www.egenix.com/ Python Software: http://www.lemburg.com/python/ From guido@python.org Fri Oct 5 16:57:53 2001 From: guido@python.org (Guido van Rossum) Date: Fri, 05 Oct 2001 11:57:53 -0400 Subject: [Python-Dev] Need a way to test for 8-bit-or-unicode-string In-Reply-To: Your message of "Fri, 05 Oct 2001 17:54:05 +0200." <3BBDD79D.C26D1646@lemburg.com> References: <200110051436.KAA24983@cj20424-a.reston1.va.home.com> <3BBDD79D.C26D1646@lemburg.com> Message-ID: <200110051557.LAA25625@cj20424-a.reston1.va.home.com> > > - Create a common base class of str and unicode, which should be an > > abstract class. This is the most OO solution, but I can't think of > > a good name; abstractstring is too long, AbstractString or String > > are uncommon naming conventions for built-in types, 'string' would > > almost work except that it's already the name of a very common > > module.[*] > > +1. This would be nice and the same could be done for sequences, file-like > objects and other common currently interface-defined object categories. > > About the naming: how about numberclass, stringclass, sequenceclass, > fileclass ?! > > Then you could write: > > if isinstance(obj, stringclass): ... > > which looks OK and is not too much typing. Hm, I don't know if "class" is the proper suffix to mean "abstract base class". I think I'll either call it "numberbase" or "abstractnumber". > The advantage of this approach is that it can be extended to other > types and classes as well (much like you can currently do with the > Python exceptions). Yup, that's why it's the most OO solution. :-) > With the new type logic in place, how hard would it be making > the existing built-in types subclasses of these base types ? That would be a prerequisite! > (also: is there a run-time penalty for this ?) None. > > - Make str a subclass of unicode (or vice versa). This can't be done > > because subclassing requires implementation inheritance, in > > particular the instance structure layout must overlap. Also, this > > would make it hard to check for either str or unicode. > > -0. This would be hard to get right because the two objects use a > very different struct layout. Could be an option in the long run though. I tried to explany that I already gave it a -1. ;-) > > - Create a new service function, IsString(x) or isString(x) or > > isstring(x), that's a shortcut for "isinstance(x, str) or > > isinstance(x, unicode)". The question them becomes where to put > > this: as a builtin, in types.py, or somewhere else... > > -1. This mechanism can not be extended by e.g. UserStrings. Good point. There's still Neil's isinstance(x, (str, unicode)) which gives the programmer more freedom: maybe some function wants to support lists and tuples but not all sequences. I'm currently +1 on introducing names for abstract base classes *and* extensing isinstance()'s API. --Guido van Rossum (home page: http://www.python.org/~guido/) From thomas.heller@ion-tof.com Fri Oct 5 17:01:41 2001 From: thomas.heller@ion-tof.com (Thomas Heller) Date: Fri, 5 Oct 2001 18:01:41 +0200 Subject: [Python-Dev] imputil and modulefinder replacements References: <3BB204E3.23339.3A9F068C@localhost> <064a01c14d84$3ed0fcf0$e000a8c0@thomasnotebook> <3BBDD254.32DC4071@interet.com> Message-ID: <085f01c14db7$06704720$e000a8c0@thomasnotebook> From: "James C. Ahlstrom" <jim@interet.com> > Thomas Heller wrote: > > > > From: "Gordon McMillan" <gmcm@hypernet.com> > > > I would like to propose these (or something quite like them) as > > > replacements for the official versions. The code is quite similar > > > (in fact, the modulefinder code could have been written by > > > subclassing the imputil stuff, but I wrote them the other way > > > 'round). > > > > > It seems noone cares about this. imputil is in the distribution, > > but is it really 'official'? > > I care about it. I am using imputil to implement "jar" files, > zip files of Python libraries. Greg worked hard to get imputil > into the distribution, and I appreciate his effort. But I always > wondered if imputil was 'official'. > > Users on c.l.p frequently want to ship a stand-alone > Python. The current answer to this is freeze, but this requires a > C compiler and some work. A standard jar file implementation would > satisfy this need. What I am thinking of is documentation which > says something like: "Zip up the Python libraries into pylibs.zip > and put that file in the same directory as python[.exe] and > everything Just Works." Do you know about py2exe and installer? This is a variation of what they do. py2exe basically builds a zip-file containing the libary modules, appends them to a special python.exe. This special exe 'boots' itself by using imputil, and exposes a 'get_code' function into the __main__ namespace. This function is called by a module name contained in the (embedded) zip, and returns the stored bytes. The installed imputil code then creates a module from these bytes. Gordon's installer works very similar, except that he uses a custom archive format (IIRC). > > My interest in imputil and ImportManager is to implement jar files, > and neither does this well. The problem is always how to get the > SpecialImporter.py imported first, and how to hack the imports that > the SpecialImporter.py may need to import. In iu4.py we see the use > of posix.stat to replace the os module, and the import of only > built-in modules sys, imp, and marshal. The same is true of imputil.py. > > I admit I don't really understand package imports. But I feel they > are an unsolved problem, because package authors always seem to > write custom ones. > > A first-class solution would have these features IMHO: > 1) Access to libraries in zip files is implemented in C (I volunteer). You could take the code from py2exe... > 2) All Python library modules can be imported by C code, > either from directories or zip files. > 3) Package import is not implemented in C, it is implemented > by a Python module which is part of the standard library. This could be imputil, or the new iu4. > 4) If an import of A.B is attempted, and a function > A.MyCustomImporter() exists, then that is used instead of > the default. > 5) More or better ideas from you all. BTW: I remember I having seen a ZipFileImporter using imputil, but cannot find it any more. Am I dreaming? Thomas From gward@python.net Fri Oct 5 17:05:35 2001 From: gward@python.net (Greg Ward) Date: Fri, 5 Oct 2001 12:05:35 -0400 Subject: [Python-Dev] Making setup.py a little more flexible In-Reply-To: <200110041712.TAA25396@pandora.informatik.hu-berlin.de> References: <200110041712.TAA25396@pandora.informatik.hu-berlin.de> Message-ID: <20011005120535.A1801@cthulhu.mems-exchange.org> [Martin von Loewis, on my band-aid patch to setup.py] > -1. First of all, a bandaid for the specific problem is already in place: > you can configure everything through Modules/Setup, in particular if you > find that setup.py doesn't do the right thing. Modules/Setup allow much > more detailed changes than what your changes to setup.py offers, plus it > is an established solution to this problem. OK, I did not realize that Modules/Setup still had its old effect. I thought that the old system was history; I guess not. Oops. In discovering this, I think I have found some bits of README that need updating. Eg. (When working inside the Modules directory, use "make Makefile; make".) -- this is untrue. There is no Modules/Makefile anymore, right? In addition to the file Setup, you can also edit the file Setup.local. (the makesetup script processes both). You may find it more convenient to edit Setup.local and leave Setup alone. Then, when installing a new Python version, you can copy your old Setup.local file. I'm not sure if this is correct either. I tried to make my changes in Setup.local -- ie. add lines to define readline and _socket, with no reference to the broken libraries on this system. It had no effect. I made this same changes in Setup, and it worked. Is this the expected behaviour? From the above paragraph, it sounds like editing Setup.local should have worked for me. > Furthermore, your specific patch seems incomplete: You have to run setup.py > manually, which means that you cannot use a plain "make". There should be > atleast minimal interfacing to Makefile.pre.in for this patch to be acceptable. Correct -- I wanted to see what sort of reaction I would get before propagating the effects of the patch upwards. It now looks like I can withdraw the patch, so I'm glad I didn't waste the time on doing that. Greg -- Greg Ward - Unix weenie gward@python.net http://starship.python.net/~gward/ "I know a lot about art, but I don't know what I like!" From guido@python.org Fri Oct 5 17:02:36 2001 From: guido@python.org (Guido van Rossum) Date: Fri, 05 Oct 2001 12:02:36 -0400 Subject: [Python-Dev] Need a way to test for 8-bit-or-unicode-string In-Reply-To: Your message of "Fri, 05 Oct 2001 11:46:48 EDT." <15293.54760.113875.151084@grendel.zope.com> References: <200110051436.KAA24983@cj20424-a.reston1.va.home.com> <20011005081136.A25134@glacier.arctrix.com> <15293.53573.251652.664773@anthem.wooz.org> <200110051546.LAA25549@cj20424-a.reston1.va.home.com> <15293.54760.113875.151084@grendel.zope.com> Message-ID: <200110051602.MAA25705@cj20424-a.reston1.va.home.com> > Guido van Rossum writes: > > Here's an implementation of Neil's idea. > > Commit this and I'll take care of the documentation. ;-) I'll wait 24 hours. Who knows what I think after the anesthetics in my jaw wear off... :-) --Guido van Rossum (home page: http://www.python.org/~guido/) From mal@lemburg.com Fri Oct 5 17:06:27 2001 From: mal@lemburg.com (M.-A. Lemburg) Date: Fri, 05 Oct 2001 18:06:27 +0200 Subject: [Python-Dev] imputil and modulefinder replacements References: <3BB204E3.23339.3A9F068C@localhost> <064a01c14d84$3ed0fcf0$e000a8c0@thomasnotebook> <3BBDD254.32DC4071@interet.com> Message-ID: <3BBDDA83.B28556CB@lemburg.com> "James C. Ahlstrom" wrote: > > Thomas Heller wrote: > > > > From: "Gordon McMillan" <gmcm@hypernet.com> > > > I would like to propose these (or something quite like them) as > > > replacements for the official versions. The code is quite similar > > > (in fact, the modulefinder code could have been written by > > > subclassing the imputil stuff, but I wrote them the other way > > > 'round). > > > > > It seems noone cares about this. imputil is in the distribution, > > but is it really 'official'? > > I care about it. I am using imputil to implement "jar" files, > zip files of Python libraries. Greg worked hard to get imputil > into the distribution, and I appreciate his effort. But I always > wondered if imputil was 'official'. > > Users on c.l.p frequently want to ship a stand-alone > Python. The current answer to this is freeze, but this requires a > C compiler and some work. A standard jar file implementation would > satisfy this need. What I am thinking of is documentation which > says something like: "Zip up the Python libraries into pylibs.zip > and put that file in the same directory as python[.exe] and > everything Just Works." > > My interest in imputil and ImportManager is to implement jar files, > and neither does this well. The problem is always how to get the > SpecialImporter.py imported first, and how to hack the imports that > the SpecialImporter.py may need to import. In iu4.py we see the use > of posix.stat to replace the os module, and the import of only > built-in modules sys, imp, and marshal. The same is true of imputil.py. > > I admit I don't really understand package imports. But I feel they > are an unsolved problem, because package authors always seem to > write custom ones. Huh ? The builtin mechanism in Python works just fine for packages. I think that custom importers are not very common these days. Still, I do agree that the current implementation is a gross hack and should in the long run be replaced by something more OO-style. For the short-term however, I have a feeling that there are not that many features actually being requested out there -- the most wanted one is certainly the ability to import modules from some form of archive, so perhaps we should start thinking about providing this kind of support in the core ?! > A first-class solution would have these features IMHO: > 1) Access to libraries in zip files is implemented in C (I volunteer). Great ! > 2) All Python library modules can be imported by C code, > either from directories or zip files. Right. > 3) Package import is not implemented in C, it is implemented > by a Python module which is part of the standard library. Why ? I don't see a need for that and it would slow down package imports which are becoming increasingly popular as the Python code base grows. > 4) If an import of A.B is attempted, and a function > A.MyCustomImporter() exists, then that is used instead of > the default. Better postpone this one... > 5) More or better ideas from you all. One question: should these ZIP-archives filenames be placed in sys.path or should Python scan for ZIP-archives within the dirs on sys.path ? -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Consulting & Company: http://www.egenix.com/ Python Software: http://www.lemburg.com/python/ From gward@python.net Fri Oct 5 17:19:31 2001 From: gward@python.net (Greg Ward) Date: Fri, 5 Oct 2001 12:19:31 -0400 Subject: [Python-Dev] imputil and modulefinder replacements In-Reply-To: <3BBDDA83.B28556CB@lemburg.com> References: <3BB204E3.23339.3A9F068C@localhost> <064a01c14d84$3ed0fcf0$e000a8c0@thomasnotebook> <3BBDD254.32DC4071@interet.com> <3BBDDA83.B28556CB@lemburg.com> Message-ID: <20011005121931.B1801@cthulhu.mems-exchange.org> On 05 October 2001, M.-A. Lemburg said: > Huh ? The builtin mechanism in Python works just fine for packages. [...] > > A first-class solution would have these features IMHO: > > 1) Access to libraries in zip files is implemented in C (I volunteer). > > Great ! > > > 2) All Python library modules can be imported by C code, > > either from directories or zip files. > > Right. > > > 3) Package import is not implemented in C, it is implemented > > by a Python module which is part of the standard library. > > Why ? I don't see a need for that and it would slow down > package imports which are becoming increasingly popular as the > Python code base grows. For the record, I agree with you on all points: there's no need to rewrite package import in Python, but it would be nice to have "import from archive" available in the core. Quite possibly adding this feature would silence 80% of the desire for genreic import hooks. The only import hook I'm familiar with is Quixote, where we have defined a Python dialect called PTL (Python Template Language, used to embed HTML [or other text] in Python). Install Quixote's import hook, and you can import .ptl files just like .py files. It's very handy. Out of curiosity, does anyone know of any other import hooks like this out there -- ie. import something that is not strictly Python. (It sounds like most import hooks/hacks deal with the location of the .py files to import. Quixote doesn't touch that, but it adds the ability to import not-quite-Python source files.) > One question: should these ZIP-archives filenames be placed in > sys.path or should Python scan for ZIP-archives within the > dirs on sys.path ? I think the archive file should be listed in sys.path. Note that my experience of this in Java was largely negative, because Java doesn't have a standard way of putting .class files in the filesystem (AFAIK) -- so everything has to be in a .jar file, and those .jar files can be anywhere you please. So you end up with a mile-long CLASSPATH that's very fragile and forever needing fixing. As long as most Python modules are accessed the ordinary way (files in a directory), then Python won't have a problem. But if somebody makes a Python installation with "stdlib.zip", "distutils.zip", "mxDateTime.zip", etc. etc., then the poor users will be in the same boat as Java users. The main concerns I have with scanning sys.path directories for ZIP files are performance and transparence. (It ain't necessarily obvious where a particular module will come from if sys.path can be searched in two ways.) Greg -- Greg Ward - Unix weenie gward@python.net http://starship.python.net/~gward/ Pointers are Arrays; Code is Data; Time is Money From akuchlin@mems-exchange.org Fri Oct 5 17:27:36 2001 From: akuchlin@mems-exchange.org (Andrew Kuchling) Date: Fri, 5 Oct 2001 12:27:36 -0400 Subject: [Python-Dev] imputil and modulefinder replacements In-Reply-To: <20011005121931.B1801@cthulhu.mems-exchange.org>; from gward@python.net on Fri, Oct 05, 2001 at 12:19:31PM -0400 References: <3BB204E3.23339.3A9F068C@localhost> <064a01c14d84$3ed0fcf0$e000a8c0@thomasnotebook> <3BBDD254.32DC4071@interet.com> <3BBDDA83.B28556CB@lemburg.com> <20011005121931.B1801@cthulhu.mems-exchange.org> Message-ID: <20011005122736.A1213@ute.mems-exchange.org> On Fri, Oct 05, 2001 at 12:19:31PM -0400, Greg Ward wrote: >can import .ptl files just like .py files. It's very handy. Out of >curiosity, does anyone know of any other import hooks like this out >there -- ie. import something that is not strictly Python. (It sounds I vaguely remember an import hook that looked for ILU or Fnorb foo.idl files on the path and automatically compiled them to foo.py if found. --amk From gward@python.net Fri Oct 5 17:28:49 2001 From: gward@python.net (Greg Ward) Date: Fri, 5 Oct 2001 12:28:49 -0400 Subject: [Python-Dev] Need a way to test for 8-bit-or-unicode-string In-Reply-To: <200110051436.KAA24983@cj20424-a.reston1.va.home.com> References: <200110051436.KAA24983@cj20424-a.reston1.va.home.com> Message-ID: <20011005122849.C1801@cthulhu.mems-exchange.org> On 05 October 2001, Guido van Rossum said: > - Create a new service function, IsString(x) or isString(x) or > isstring(x), that's a shortcut for "isinstance(x, str) or > isinstance(x, unicode)". The question them becomes where to put > this: as a builtin, in types.py, or somewhere else... I've thought Python should have this for a long time. But Marc-Andre's point pretty much shoots it down, unless you make isstring(x) sugar for isinstance(x, (str, unicode)) Hmmm. It's a very small dose of sugar, but I like it all the same. > [...] (BTW this shows to me again that the method > signature is right and the function signature is wrong. But even my > time machine isn't powerful enough to fix this.) (Hm, it could be > saved by making string.join() accept the arguments in either order. > Gross. :-) Note that a certain infamous April Fool's post from a few years back said that that was what Python 1.6 would do. I wasn't serious, either. ;-) Greg -- Greg Ward - just another /P(erl|ython)/ hacker gward@python.net http://starship.python.net/~gward/ LILO boot: linux init=/usr/bin/emacs From barry@zope.com Fri Oct 5 17:30:34 2001 From: barry@zope.com (Barry A. Warsaw) Date: Fri, 5 Oct 2001 12:30:34 -0400 Subject: [Python-Dev] Need a way to test for 8-bit-or-unicode-string References: <200110051436.KAA24983@cj20424-a.reston1.va.home.com> <3BBDD79D.C26D1646@lemburg.com> <200110051557.LAA25625@cj20424-a.reston1.va.home.com> Message-ID: <15293.57386.311536.729065@anthem.wooz.org> >>>>> "GvR" == Guido van Rossum <guido@python.org> writes: GvR> I'm currently +1 on introducing names for abstract base GvR> classes *and* extensing isinstance()'s API. I'm worried about the abstract base class idea, because it might lead us down a path of having a complicated class hierarchy in order to provide completeness. E.g. should "strings" be a sequencebase, as well as a stringbase? And what does a sequencebase /mean/? Will we push to try to include protocols/interfaces into this mix, so that we'll end up defining abstract classes for all the deep-mojo interfaces? Maybe we'll want to have abstract base classes for file objects so things like file and StringIO's can have them as a common base class? In any event, I think the isinstance() extension is a simple, clean, and (mostly :) uncontroversial change, so I'm also +1 on doing that now. Adding a set of abstract base classes and hooking them up with the existing concrete data types seems much murkier to me, and of greater long range impact, so I'm -1 on the idea, at least until it gets PEP'd. -Barry From jeremy@zope.com Fri Oct 5 17:31:30 2001 From: jeremy@zope.com (Jeremy Hylton) Date: Fri, 5 Oct 2001 12:31:30 -0400 (EDT) Subject: [Python-Dev] Need a way to test for 8-bit-or-unicode-string In-Reply-To: <200110051557.LAA25625@cj20424-a.reston1.va.home.com> References: <200110051436.KAA24983@cj20424-a.reston1.va.home.com> <3BBDD79D.C26D1646@lemburg.com> <200110051557.LAA25625@cj20424-a.reston1.va.home.com> Message-ID: <15293.57442.799246.716550@slothrop.digicool.com> >>>>> "GvR" == Guido van Rossum <guido@python.org> writes: GvR> There's still Neil's isinstance(x, (str, unicode)) which gives GvR> the programmer more freedom: maybe some function wants to GvR> support lists and tuples but not all sequences. GvR> I'm currently +1 on introducing names for abstract base classes GvR> *and* extensing isinstance()'s API. +1. Jeremy From nas@python.ca Fri Oct 5 17:42:09 2001 From: nas@python.ca (Neil Schemenauer) Date: Fri, 5 Oct 2001 09:42:09 -0700 Subject: [Python-Dev] Need a way to test for 8-bit-or-unicode-string In-Reply-To: <200110051546.LAA25549@cj20424-a.reston1.va.home.com>; from guido@python.org on Fri, Oct 05, 2001 at 11:46:57AM -0400 References: <200110051436.KAA24983@cj20424-a.reston1.va.home.com> <20011005081136.A25134@glacier.arctrix.com> <15293.53573.251652.664773@anthem.wooz.org> <200110051546.LAA25549@cj20424-a.reston1.va.home.com> Message-ID: <20011005094209.A25437@glacier.arctrix.com> Guido van Rossum wrote: > Here's an implementation of Neil's idea. > > *** abstract.c 2001/10/01 17:10:18 2.83 > --- abstract.c 2001/10/05 15:49:04 > *************** > *** 1805,1810 **** > --- 1805,1829 ---- > else if (PyType_Check(cls)) { > retval = PyObject_TypeCheck(inst, (PyTypeObject *)cls); > } > + else if (PySequence_Check(cls)) { > + int i, n; Is is possible that a type also passes PySequence_Check? If so, that could lead to confusing behavior. Neil From guido@python.org Fri Oct 5 17:50:48 2001 From: guido@python.org (Guido van Rossum) Date: Fri, 05 Oct 2001 12:50:48 -0400 Subject: [Python-Dev] Need a way to test for 8-bit-or-unicode-string In-Reply-To: Your message of "Fri, 05 Oct 2001 09:42:09 PDT." <20011005094209.A25437@glacier.arctrix.com> References: <200110051436.KAA24983@cj20424-a.reston1.va.home.com> <20011005081136.A25134@glacier.arctrix.com> <15293.53573.251652.664773@anthem.wooz.org> <200110051546.LAA25549@cj20424-a.reston1.va.home.com> <20011005094209.A25437@glacier.arctrix.com> Message-ID: <200110051650.MAA26943@cj20424-a.reston1.va.home.com> > > *** 1805,1810 **** > > --- 1805,1829 ---- > > else if (PyType_Check(cls)) { > > retval = PyObject_TypeCheck(inst, (PyTypeObject *)cls); > > } > > + else if (PySequence_Check(cls)) { > > + int i, n; > > Is is possible that a type also passes PySequence_Check? If so, that > could lead to confusing behavior. The only way a type could pass PySequence_Check is if someone subclasses 'type' and adds a __getitem__ method. In that case, it's appropriate that PyType_Check() has prevalence. Also, adding __getitem__ to a type seems to serve no purpose, so I'm not worried about this (and yes, I had thought about this :-). --Guido van Rossum (home page: http://www.python.org/~guido/) From fdrake@acm.org Fri Oct 5 17:51:22 2001 From: fdrake@acm.org (Fred L. Drake) Date: Fri, 5 Oct 2001 12:51:22 -0400 (EDT) Subject: [Python-Dev] [development doc updates] Message-ID: <20011005165122.33A8A28697@beowolf.digicool.com> The development version of the documentation has been updated: http://python.sourceforge.net/devel-docs/ Removed reference material for some long-obsolete Mac OS modules that have not been included in recent versions of MacPython. Added material on the proposed C API for weak reference objects; see the Python/C API Reference Manual: http://python.sourceforge.net/devel-docs/api/weakref-objects.html From mal@lemburg.com Fri Oct 5 17:57:20 2001 From: mal@lemburg.com (M.-A. Lemburg) Date: Fri, 05 Oct 2001 18:57:20 +0200 Subject: [Python-Dev] Need a way to test for 8-bit-or-unicode-string References: <200110051436.KAA24983@cj20424-a.reston1.va.home.com> <3BBDD79D.C26D1646@lemburg.com> <200110051557.LAA25625@cj20424-a.reston1.va.home.com> <15293.57386.311536.729065@anthem.wooz.org> Message-ID: <3BBDE670.52DFB7F0@lemburg.com> "Barry A. Warsaw" wrote: > > >>>>> "GvR" == Guido van Rossum <guido@python.org> writes: > > GvR> I'm currently +1 on introducing names for abstract base > GvR> classes *and* extensing isinstance()'s API. > > I'm worried about the abstract base class idea, because it might lead > us down a path of having a complicated class hierarchy in order to > provide completeness. Since these are abstract classes (basically names for what we now loosly call interface, e.g. file-like object, sequence-like object etc.) this very shallow hierarchy wouldn't hurt all that much. Sure, strings are sequences, so they stringbase would have to be a subclass of sequencebase, but I don't think we'll ever get more than 3 levels deep in the abstract class hierarchy. > E.g. should "strings" be a sequencebase, as well as a stringbase? And > what does a sequencebase /mean/? Good point. There's PySequence_Check(), but we'll have to discuss this at some point anyway and come up with definitions for the currently used fuzzy terms "sequence", "number", "string", and so on. > Will we push to try to include protocols/interfaces into this mix, so > that we'll end up defining abstract classes for all the deep-mojo > interfaces? Maybe we'll want to have abstract base classes for file > objects so things like file and StringIO's can have them as a common > base class? > > In any event, I think the isinstance() extension is a simple, clean, > and (mostly :) uncontroversial change, so I'm also +1 on doing that > now. > > Adding a set of abstract base classes and hooking them up with the > existing concrete data types seems much murkier to me, and of greater > long range impact, so I'm -1 on the idea, at least until it gets > PEP'd. Are you sure about the -1 ? Using abstract base classes seems like a natural thing to do in an OO-language like Python while the isinstance() hack would not allow you to use your own UserList subclass for code which was written with sequences as input. -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Consulting & Company: http://www.egenix.com/ Python Software: http://www.lemburg.com/python/ From guido@python.org Fri Oct 5 17:55:31 2001 From: guido@python.org (Guido van Rossum) Date: Fri, 05 Oct 2001 12:55:31 -0400 Subject: [Python-Dev] Need a way to test for 8-bit-or-unicode-string In-Reply-To: Your message of "Fri, 05 Oct 2001 12:30:34 EDT." <15293.57386.311536.729065@anthem.wooz.org> References: <200110051436.KAA24983@cj20424-a.reston1.va.home.com> <3BBDD79D.C26D1646@lemburg.com> <200110051557.LAA25625@cj20424-a.reston1.va.home.com> <15293.57386.311536.729065@anthem.wooz.org> Message-ID: <200110051655.MAA26969@cj20424-a.reston1.va.home.com> > GvR> I'm currently +1 on introducing names for abstract base > GvR> classes *and* extensing isinstance()'s API. > > I'm worried about the abstract base class idea, because it might lead > us down a path of having a complicated class hierarchy in order to > provide completeness. Yes, that's a danger. I'm currently thinking about adding a small number of abstract base classes, corresponding to the groups of basic concrete data types in Python: numberbase, integerbase, sequencebase, stringbase, mappingbase, filebase. And I'm not even sure of these! Practically, I only see a use for stringbase and integerbase, since these have two concrete subclasses; the others are just to make the OO zealots happy. :-) > E.g. should "strings" be a sequencebase, as well as a stringbase? Yes. Any way you can think of that checks for a "sequence" will consider strings to be sequences, unless you make explicit exceptions. > And what does a sequencebase /mean/? Whatever I want it to mean. :-) The latest code for sequence-ness tests for sq_getitem != NULL, and I think that's reasonable. > Will we push to try to include protocols/interfaces into this mix, so > that we'll end up defining abstract classes for all the deep-mojo > interfaces? I'd prefer not to. > Maybe we'll want to have abstract base classes for file > objects so things like file and StringIO's can have them as a common > base class? That's currently hard. I don't think we can make StringIO a new-style class, and classic classes can't inherit from new-style classes (not even abstract ones). > In any event, I think the isinstance() extension is a simple, clean, > and (mostly :) uncontroversial change, so I'm also +1 on doing that > now. Yes. > Adding a set of abstract base classes and hooking them up with the > existing concrete data types seems much murkier to me, and of greater > long range impact, so I'm -1 on the idea, at least until it gets > PEP'd. I'm at best +0 myself -- I have more thinking to do... I've *mostly* convinced myself that stringbase is a good idea. But even there, the existence of the buffer API makes the semantics slightly murky. --Guido van Rossum (home page: http://www.python.org/~guido/) From barry@zope.com Fri Oct 5 18:04:39 2001 From: barry@zope.com (Barry A. Warsaw) Date: Fri, 5 Oct 2001 13:04:39 -0400 Subject: [Python-Dev] Need a way to test for 8-bit-or-unicode-string References: <200110051436.KAA24983@cj20424-a.reston1.va.home.com> <3BBDD79D.C26D1646@lemburg.com> <200110051557.LAA25625@cj20424-a.reston1.va.home.com> <15293.57386.311536.729065@anthem.wooz.org> <3BBDE670.52DFB7F0@lemburg.com> Message-ID: <15293.59431.1516.201622@anthem.wooz.org> >>>>> "M" == M <mal@lemburg.com> writes: M> Are you sure about the -1 ? Using abstract base classes seems M> like a natural thing to do in an OO-language like Python while M> the isinstance() hack would not allow you to use your own M> UserList subclass for code which was written with sequences as M> input. Yes, I'm sure: we need a PEP first. Ideally it would answer some of the questions raised above, and it would lay out the class hierarchy. -Barry From guido@python.org Fri Oct 5 18:02:07 2001 From: guido@python.org (Guido van Rossum) Date: Fri, 05 Oct 2001 13:02:07 -0400 Subject: [Python-Dev] Making setup.py a little more flexible In-Reply-To: Your message of "Fri, 05 Oct 2001 12:05:35 EDT." <20011005120535.A1801@cthulhu.mems-exchange.org> References: <200110041712.TAA25396@pandora.informatik.hu-berlin.de> <20011005120535.A1801@cthulhu.mems-exchange.org> Message-ID: <200110051702.NAA27256@cj20424-a.reston1.va.home.com> > In discovering this, I think I have found some bits of README that need > updating. Eg. > > (When working inside the Modules directory, use "make Makefile; > make".) > > -- this is untrue. There is no Modules/Makefile anymore, right? Correct. Fixed. > In addition to the file Setup, you can also edit the file Setup.local. > (the makesetup script processes both). You may find it more > convenient to edit Setup.local and leave Setup alone. Then, when > installing a new Python version, you can copy your old Setup.local > file. > > I'm not sure if this is correct either. I tried to make my changes in > Setup.local -- ie. add lines to define readline and _socket, with no > reference to the broken libraries on this system. It had no effect. I > made this same changes in Setup, and it worked. Is this the expected > behaviour? From the above paragraph, it sounds like editing Setup.local > should have worked for me. Strange. When I make changes to Setup.local, they definitely work! --Guido van Rossum (home page: http://www.python.org/~guido/) From fdrake@acm.org Fri Oct 5 18:01:29 2001 From: fdrake@acm.org (Fred L. Drake, Jr.) Date: Fri, 5 Oct 2001 13:01:29 -0400 Subject: [Python-Dev] Need a way to test for 8-bit-or-unicode-string In-Reply-To: <15293.59431.1516.201622@anthem.wooz.org> References: <200110051436.KAA24983@cj20424-a.reston1.va.home.com> <3BBDD79D.C26D1646@lemburg.com> <200110051557.LAA25625@cj20424-a.reston1.va.home.com> <15293.57386.311536.729065@anthem.wooz.org> <3BBDE670.52DFB7F0@lemburg.com> <15293.59431.1516.201622@anthem.wooz.org> Message-ID: <15293.59241.954894.415782@grendel.zope.com> Barry A. Warsaw writes: > Yes, I'm sure: we need a PEP first. Ideally it would answer some of > the questions raised above, and it would lay out the class hierarchy. Sounds to me like you just volunteered. And on that note, its time for some cheap Chinese food... -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> PythonLabs at Zope Corporation From guido@python.org Fri Oct 5 18:07:32 2001 From: guido@python.org (Guido van Rossum) Date: Fri, 05 Oct 2001 13:07:32 -0400 Subject: [Python-Dev] Need a way to test for 8-bit-or-unicode-string In-Reply-To: Your message of "Fri, 05 Oct 2001 13:04:39 EDT." <15293.59431.1516.201622@anthem.wooz.org> References: <200110051436.KAA24983@cj20424-a.reston1.va.home.com> <3BBDD79D.C26D1646@lemburg.com> <200110051557.LAA25625@cj20424-a.reston1.va.home.com> <15293.57386.311536.729065@anthem.wooz.org> <3BBDE670.52DFB7F0@lemburg.com> <15293.59431.1516.201622@anthem.wooz.org> Message-ID: <200110051707.NAA27421@cj20424-a.reston1.va.home.com> > Yes, I'm sure: we need a PEP first. Ideally it would answer some of > the questions raised above, and it would lay out the class hierarchy. The more I think about this, the more I agree that we need a PEP first: there are lots of potential traps. Maybe the biggest danger: declaring an abstract base class for e.g. "sequence-ness" would encourage programmers to test isinstance(x, sequencebase) when all they *really* need is something that supports the __getitem__ protocol. This would encourage writing code that is less polymorphic than it could be, and that would be a real loss. --Guido van Rossum (home page: http://www.python.org/~guido/) From barry@zope.com Fri Oct 5 18:18:17 2001 From: barry@zope.com (Barry A. Warsaw) Date: Fri, 5 Oct 2001 13:18:17 -0400 Subject: [Python-Dev] Need a way to test for 8-bit-or-unicode-string References: <200110051436.KAA24983@cj20424-a.reston1.va.home.com> <3BBDD79D.C26D1646@lemburg.com> <200110051557.LAA25625@cj20424-a.reston1.va.home.com> <15293.57386.311536.729065@anthem.wooz.org> <3BBDE670.52DFB7F0@lemburg.com> <15293.59431.1516.201622@anthem.wooz.org> <15293.59241.954894.415782@grendel.zope.com> Message-ID: <15293.60249.661659.388298@anthem.wooz.org> >>>>> "Fred" == Fred L Drake, Jr <fdrake@acm.org> writes: >> Yes, I'm sure: we need a PEP first. Ideally it would answer >> some of the questions raised above, and it would lay out the >> class hierarchy. Fred> Sounds to me like you just volunteered. Fred> And on that note, its time for some cheap Chinese food... I think the drop in your BKPL (blood Kung Pao level) has affected your hearing. Please, hurry to boost it back up! sound-of-one-chopstick-clapping-ly y'rs, -Barry From mclay@nist.gov Fri Oct 5 18:28:06 2001 From: mclay@nist.gov (Michael McLay) Date: Fri, 5 Oct 2001 13:28:06 -0400 Subject: [Python-Dev] imputil and modulefinder replacements In-Reply-To: <20011005121931.B1801@cthulhu.mems-exchange.org> References: <3BB204E3.23339.3A9F068C@localhost> <3BBDDA83.B28556CB@lemburg.com> <20011005121931.B1801@cthulhu.mems-exchange.org> Message-ID: <0110051328060C.02197@fermi.eeel.nist.gov> On Friday 05 October 2001 12:19 pm, Greg Ward wrote: > On 05 October 2001, M.-A. Lemburg said: > The only import hook I'm familiar with is Quixote, where we have defined > a Python dialect called PTL (Python Template Language, used to embed > HTML [or other text] in Python). Install Quixote's import hook, and you > can import .ptl files just like .py files. It's very handy. Out of > curiosity, does anyone know of any other import hooks like this out > there -- ie. import something that is not strictly Python. The dpython interpreter I wrote this summer would tokenize .dp files that were in the Python path. If the alternative suffix was detected the Python interpreter would use the dpython tokenizer. The dpython tokenizer would generate decimal numbers instead of binary numbers as the default for float and int literals. I didn't use the import hooks. I built the .dp file recognition into the interpreter at the point where it evaluates the suffix of the file name and compairs it to the filedesc struct. When I discussed the implementation of dpython at the ZPUG meeting I joked about extending the concept to add bytecode generators for JavaScript and Visual Basic modules. The work required to add JavaScript would probably be reasonable. It might help get Python used anywhere JavaScript is used. There has also been some interest at NIST and NASA in building modules to parse meta languages such as XML Schema, KIF, and Express and directly generate Python bytecodes. From pedroni@inf.ethz.ch Fri Oct 5 19:06:23 2001 From: pedroni@inf.ethz.ch (Samuele Pedroni) Date: Fri, 5 Oct 2001 20:06:23 +0200 Subject: [Python-Dev] imputil and modulefinder replacements References: <3BB204E3.23339.3A9F068C@localhost> <064a01c14d84$3ed0fcf0$e000a8c0@thomasnotebook> <3BBDD254.32DC4071@interet.com> <3BBDDA83.B28556CB@lemburg.com> <20011005121931.B1801@cthulhu.mems-exchange.org> Message-ID: <001101c14dc8$754468a0$8a73fea9@newmexico> Hi [Greg Ward] > > One question: should these ZIP-archives filenames be placed in > > sys.path or should Python scan for ZIP-archives within the > > dirs on sys.path ? > > I think the archive file should be listed in sys.path. the latest jython code base (2.1a3) allow to put jar/zip in the sys.path or reference of the form jarfile!pkgdir1/pkgdir2 This are used to set meaningful __path__ attrs for the packages and should allow to deal with code that change __path__ pointing to some relative subdir according to some context property. > Note that my experience of this in Java was largely negative, because > Java doesn't have a standard way of putting .class files in the > filesystem (AFAIK) -- so everything has to be in a .jar file, and those > .jar files can be anywhere you please. I don't get the point :( > So you end up with a mile-long > CLASSPATH that's very fragile and forever needing fixing. As long as > most Python modules are accessed the ordinary way (files in a > directory), then Python won't have a problem. But if somebody makes a > Python installation with "stdlib.zip", "distutils.zip", > "mxDateTime.zip", etc. etc., then the poor users will be in the same > boat as Java users. > > The main concerns I have with scanning sys.path directories for ZIP > files are performance and transparence. (It ain't necessarily obvious > where a particular module will come from if sys.path can be searched in > two ways.) I agree. regards. From mal@lemburg.com Fri Oct 5 19:21:29 2001 From: mal@lemburg.com (M.-A. Lemburg) Date: Fri, 05 Oct 2001 20:21:29 +0200 Subject: [Python-Dev] Need a way to test for 8-bit-or-unicode-string References: <200110051436.KAA24983@cj20424-a.reston1.va.home.com> <3BBDD79D.C26D1646@lemburg.com> <200110051557.LAA25625@cj20424-a.reston1.va.home.com> <15293.57386.311536.729065@anthem.wooz.org> <3BBDE670.52DFB7F0@lemburg.com> <15293.59431.1516.201622@anthem.wooz.org> Message-ID: <3BBDFA29.A5A4DCBA@lemburg.com> "Barry A. Warsaw" wrote: > > >>>>> "M" == M <mal@lemburg.com> writes: > > M> Are you sure about the -1 ? Using abstract base classes seems > M> like a natural thing to do in an OO-language like Python while > M> the isinstance() hack would not allow you to use your own > M> UserList subclass for code which was written with sequences as > M> input. > > Yes, I'm sure: we need a PEP first. Ideally it would answer some of > the questions raised above, and it would lay out the class hierarchy. Maybe I wasn't clear enough: Of course we need a PEP for these names, their intended meaning as object category abstraction and interface placeholder. Too late for Python 2.2, but probably a nice project for 2.3. -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Consulting & Company: http://www.egenix.com/ Python Software: http://www.lemburg.com/python/ From jim@interet.com Fri Oct 5 19:47:31 2001 From: jim@interet.com (James C. Ahlstrom) Date: Fri, 05 Oct 2001 14:47:31 -0400 Subject: [Python-Dev] imputil and modulefinder replacements References: <3BB204E3.23339.3A9F068C@localhost> <064a01c14d84$3ed0fcf0$e000a8c0@thomasnotebook> <3BBDD254.32DC4071@interet.com> <3BBDDA83.B28556CB@lemburg.com> <20011005121931.B1801@cthulhu.mems-exchange.org> Message-ID: <3BBE0043.663D0411@interet.com> Greg Ward wrote: > > On 05 October 2001, M.-A. Lemburg said: > > One question: should these ZIP-archives filenames be placed in > > sys.path or should Python scan for ZIP-archives within the > > dirs on sys.path ? > > I think the archive file should be listed in sys.path. I think that is correct. At one time there was talk of sys.path containing importer instances. But currently (please check me here) sys.path is purely strings, and these strings must be directory names. Python depends on directories and subdirectories for its import semantics. So any zip archive must support subdirectories to support general imports. Therefore, a string in sys.path which is a zip file is equivalent to hanging a nameless subdirectory at that point. "Nameless" means zip archive files *.pyc occur at the top level, and the zip file may contain relative subdirectories with files sub1/file1.pyc, etc. So a zip file can always substitute for an arbitrary subdirectory tree, and is functionally equivalent to a directory name at the same place in sys.path. JimA From thomas.heller@ion-tof.com Fri Oct 5 19:51:19 2001 From: thomas.heller@ion-tof.com (Thomas Heller) Date: Fri, 5 Oct 2001 20:51:19 +0200 Subject: [Python-Dev] threading bug? Message-ID: <0ac401c14dce$b889b9c0$e000a8c0@thomasnotebook> [Apologies, I also asked this on c.l.p, but got no answer so far. I have not too much time left before I will be away for nearly 2 weeks] I wonder if this is a bug. I'm running a deamon thread doing some work in the background, and sometimes when Python exits I get tracebacks like this: Unhandled exception in thread: Traceback (most recent call last): File "c:\python21\lib\threading.py", line 393, in __bootstrap self.__stop() File "c:\python21\lib\threading.py", line 399, in __stop self.__block.notifyAll() File "c:\python21\lib\threading.py", line 235, in notifyAll self.notify(len(self.__waiters)) File "c:\python21\lib\threading.py", line 217, in notify me = currentThread() TypeError: object of type 'None' is not callable This traceback does not appear when I register an atexit function to stop the thread. Is this a bug? Aren't daemon threads supposed to clean up themselves? Thomas From guido@python.org Fri Oct 5 19:54:26 2001 From: guido@python.org (Guido van Rossum) Date: Fri, 05 Oct 2001 14:54:26 -0400 Subject: [Python-Dev] threading bug? In-Reply-To: Your message of "Fri, 05 Oct 2001 20:51:19 +0200." <0ac401c14dce$b889b9c0$e000a8c0@thomasnotebook> References: <0ac401c14dce$b889b9c0$e000a8c0@thomasnotebook> Message-ID: <200110051854.OAA28426@cj20424-a.reston1.va.home.com> > [Apologies, I also asked this on c.l.p, but got no > answer so far. I have not too much time left before > I will be away for nearly 2 weeks] > > I wonder if this is a bug. I'm running a deamon thread doing > some work in the background, and sometimes when Python exits > I get tracebacks like this: > > Unhandled exception in thread: > Traceback (most recent call last): > File "c:\python21\lib\threading.py", line 393, in __bootstrap > self.__stop() > File "c:\python21\lib\threading.py", line 399, in __stop > self.__block.notifyAll() > File "c:\python21\lib\threading.py", line 235, in notifyAll > self.notify(len(self.__waiters)) > File "c:\python21\lib\threading.py", line 217, in notify > me = currentThread() > TypeError: object of type 'None' is not callable > > This traceback does not appear when I register an atexit > function to stop the thread. > > Is this a bug? Aren't daemon threads supposed to clean up > themselves? This is probably a result of the destructive module finalization. When Py_Finalize() is called, threads that are still running are not killed. But Py_Finalize() replaces the values of all globals in modules with None (it could just clear the module's dictionary but there was some reason for doing it this way) and that is what you are probably seeing. If there was a primitive to kill a thread, we might use it in Py_Finalize() -- but there isn't (it's been requested but it's hard to see how to do it safely). --Guido van Rossum (home page: http://www.python.org/~guido/) From jim@interet.com Fri Oct 5 19:58:44 2001 From: jim@interet.com (James C. Ahlstrom) Date: Fri, 05 Oct 2001 14:58:44 -0400 Subject: [Python-Dev] imputil and modulefinder replacements References: <3BB204E3.23339.3A9F068C@localhost> <064a01c14d84$3ed0fcf0$e000a8c0@thomasnotebook> <3BBDD254.32DC4071@interet.com> <085f01c14db7$06704720$e000a8c0@thomasnotebook> Message-ID: <3BBE02E4.555D64BF@interet.com> Thomas Heller wrote: > Do you know about py2exe and installer? This is a variation of what they do. I've heard of it, but where is it? I never searched out the URL. I just use freeze. Anyway, I think we something in the core. > BTW: I remember I having seen a ZipFileImporter using imputil, but > cannot find it any more. Am I dreaming? Either Greg Stein or I (I don't remember) wrote a ZipFileImporter for imputil, and that is what I am currently using. Jim From thomas.heller@ion-tof.com Fri Oct 5 20:04:00 2001 From: thomas.heller@ion-tof.com (Thomas Heller) Date: Fri, 5 Oct 2001 21:04:00 +0200 Subject: [Python-Dev] threading bug? References: <0ac401c14dce$b889b9c0$e000a8c0@thomasnotebook> <200110051854.OAA28426@cj20424-a.reston1.va.home.com> Message-ID: <0afd01c14dd0$7e170a20$e000a8c0@thomasnotebook> From: "Guido van Rossum" <guido@python.org> > > [Apologies, I also asked this on c.l.p, but got no > > answer so far. I have not too much time left before > > I will be away for nearly 2 weeks] > > > > I wonder if this is a bug. I'm running a deamon thread doing > > some work in the background, and sometimes when Python exits > > I get tracebacks like this: > > > > Unhandled exception in thread: > > Traceback (most recent call last): > > File "c:\python21\lib\threading.py", line 393, in __bootstrap > > self.__stop() > > File "c:\python21\lib\threading.py", line 399, in __stop > > self.__block.notifyAll() > > File "c:\python21\lib\threading.py", line 235, in notifyAll > > self.notify(len(self.__waiters)) > > File "c:\python21\lib\threading.py", line 217, in notify > > me = currentThread() > > TypeError: object of type 'None' is not callable > > > > This traceback does not appear when I register an atexit > > function to stop the thread. > > > > Is this a bug? Aren't daemon threads supposed to clean up > > themselves? > > This is probably a result of the destructive module finalization. > When Py_Finalize() is called, threads that are still running are not > killed. But Py_Finalize() replaces the values of all globals in > modules with None (it could just clear the module's dictionary but > there was some reason for doing it this way) and that is what you are > probably seeing. > > If there was a primitive to kill a thread, we might use it in > Py_Finalize() -- but there isn't (it's been requested but it's hard to > see how to do it safely). But it seems the system is trying to stop the thread(s), maybe too late. In my case, atexit.register(self.stop) makes a clean shutdown. My point is: should the threading module do this? I can easily live with doing it myself - I just don't know if this is intended or required. Thomas From gward@python.net Fri Oct 5 20:07:04 2001 From: gward@python.net (Greg Ward) Date: Fri, 5 Oct 2001 15:07:04 -0400 Subject: [Python-Dev] Making setup.py a little more flexible In-Reply-To: <200110051702.NAA27256@cj20424-a.reston1.va.home.com> References: <200110041712.TAA25396@pandora.informatik.hu-berlin.de> <20011005120535.A1801@cthulhu.mems-exchange.org> <200110051702.NAA27256@cj20424-a.reston1.va.home.com> Message-ID: <20011005150704.A2640@cthulhu.mems-exchange.org> On 05 October 2001, Guido van Rossum said: > Strange. When I make changes to Setup.local, they definitely work! OK, I'll try to investigate more next week. Off for Thanksgiving long weekend now! Greg -- Greg Ward - geek gward@python.net http://starship.python.net/~gward/ No animals were harmed in transmitting this message. From thomas.heller@ion-tof.com Fri Oct 5 20:09:58 2001 From: thomas.heller@ion-tof.com (Thomas Heller) Date: Fri, 5 Oct 2001 21:09:58 +0200 Subject: [Python-Dev] imputil and modulefinder replacements References: <3BB204E3.23339.3A9F068C@localhost> <064a01c14d84$3ed0fcf0$e000a8c0@thomasnotebook> <3BBDD254.32DC4071@interet.com> <085f01c14db7$06704720$e000a8c0@thomasnotebook> <3BBE02E4.555D64BF@interet.com> Message-ID: <0b0501c14dd1$539d83e0$e000a8c0@thomasnotebook> From: "James C. Ahlstrom" <jim@interet.com> > Thomas Heller wrote: > > > Do you know about py2exe and installer? This is a variation of what they do. > > I've heard of it, but where is it? I never searched out the URL. > I just use freeze. Anyway, I think we something in the core. http://starship.python.net/crew/theller/py2exe, the project is on sourceforge: http://sourceforge.net/projects/py2exe, so you can browse the CVS repository. Installer: http://www.mcmillan-inc.com/install1.html Um, BTW: py2exe's code originated from bdist_wininst (the distutils windows installer), which also contains C-code to unpack zip-files: http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/python/distutils/misc/ Thomas From guido@python.org Fri Oct 5 20:09:43 2001 From: guido@python.org (Guido van Rossum) Date: Fri, 05 Oct 2001 15:09:43 -0400 Subject: [Python-Dev] threading bug? In-Reply-To: Your message of "Fri, 05 Oct 2001 21:04:00 +0200." <0afd01c14dd0$7e170a20$e000a8c0@thomasnotebook> References: <0ac401c14dce$b889b9c0$e000a8c0@thomasnotebook> <200110051854.OAA28426@cj20424-a.reston1.va.home.com> <0afd01c14dd0$7e170a20$e000a8c0@thomasnotebook> Message-ID: <200110051909.PAA28498@cj20424-a.reston1.va.home.com> > > > Unhandled exception in thread: > > > Traceback (most recent call last): > > > File "c:\python21\lib\threading.py", line 393, in __bootstrap > > > self.__stop() > > > File "c:\python21\lib\threading.py", line 399, in __stop > > > self.__block.notifyAll() > > > File "c:\python21\lib\threading.py", line 235, in notifyAll > > > self.notify(len(self.__waiters)) > > > File "c:\python21\lib\threading.py", line 217, in notify > > > me = currentThread() > > > TypeError: object of type 'None' is not callable > > > > > > This traceback does not appear when I register an atexit > > > function to stop the thread. > > > > > > Is this a bug? Aren't daemon threads supposed to clean up > > > themselves? > > This is probably a result of the destructive module finalization. > > When Py_Finalize() is called, threads that are still running are not > > killed. But Py_Finalize() replaces the values of all globals in > > modules with None (it could just clear the module's dictionary but > > there was some reason for doing it this way) and that is what you are > > probably seeing. > > > > If there was a primitive to kill a thread, we might use it in > > Py_Finalize() -- but there isn't (it's been requested but it's hard to > > see how to do it safely). > But it seems the system is trying to stop the thread(s), > maybe too late. In my case, atexit.register(self.stop) > makes a clean shutdown. > My point is: should the threading module do this? > I can easily live with doing it myself - I just don't know > if this is intended or required. Python doesn't attempt to stop daemon threads at all. Note that the __stop() call is in a finally clause. This probably gets hit because some other exception happened (caused by the same problem of globals being replaced by None). --Guido van Rossum (home page: http://www.python.org/~guido/) From jim@interet.com Fri Oct 5 20:13:44 2001 From: jim@interet.com (James C. Ahlstrom) Date: Fri, 05 Oct 2001 15:13:44 -0400 Subject: [Python-Dev] imputil and modulefinder replacements References: <3BB204E3.23339.3A9F068C@localhost> <064a01c14d84$3ed0fcf0$e000a8c0@thomasnotebook> <3BBDD254.32DC4071@interet.com> <3BBDDA83.B28556CB@lemburg.com> Message-ID: <3BBE0668.4DF5038C@interet.com> "M.-A. Lemburg" wrote: > > "James C. Ahlstrom" wrote: > > 3) Package import is not implemented in C, it is implemented > > by a Python module which is part of the standard library. > > Why ? I don't see a need for that and it would slow down > package imports which are becoming increasingly popular as the > Python code base grows. Well, I looked at import.c and adding zip files with subdirectories will be a mess. The path searches would have to be extended to zip files. Moving out the package code would be a great simplification. Or maybe I am lazy. It doesn't help that I don't really understand packages. Your point about efficiency is a good one though, but I am not sure of the magnitude of the slow down. Anyone else have an opinion? JimA From skip@pobox.com (Skip Montanaro) Fri Oct 5 20:13:49 2001 From: skip@pobox.com (Skip Montanaro) (Skip Montanaro) Date: Fri, 5 Oct 2001 14:13:49 -0500 Subject: [Python-Dev] os.spawnv[e] and related constants Message-ID: <15294.1645.63995.884978@beluga.mojam.com> The spawnv and spawnve functions in the os module look useful, but the lib manual documentation is rather sparse about their semantics, referring the reader to the VC++ Runtime Library for more info. Perhaps this is poetic justice for us Unix bigots (since many people often complain that the Python library and docs are too Unix-centric), but it would be useful to know what the return values (if any) of these functions are and what the related P_* constants mean. I can guess about P_WAIT and P_NOWAIT, but the meanings of the others aren't obvious to a person without Windows experience. Skip From tim.one@home.com Fri Oct 5 20:16:29 2001 From: tim.one@home.com (Tim Peters) Date: Fri, 5 Oct 2001 15:16:29 -0400 Subject: [Python-Dev] threading bug? In-Reply-To: <0ac401c14dce$b889b9c0$e000a8c0@thomasnotebook> Message-ID: <LNBBLJKPBEHFEDALKOLCOEIFLPAA.tim.one@home.com> [Thomas Heller] > Unhandled exception in thread: > Traceback (most recent call last): > File "c:\python21\lib\threading.py", line 393, in __bootstrap > self.__stop() > File "c:\python21\lib\threading.py", line 399, in __stop > self.__block.notifyAll() > File "c:\python21\lib\threading.py", line 235, in notifyAll > self.notify(len(self.__waiters)) > File "c:\python21\lib\threading.py", line 217, in notify > me = currentThread() > TypeError: object of type 'None' is not callable > > This traceback does not appear when I register an atexit > function to stop the thread. We're missing basic info, like which OS and which Python release. More than one shutdown race has been fixed over the decades <wink>. I've seen reports of this before, but under older Pythons, and/or in conjunction with running GUIs where the problem went away if the GUI was taken out of the equation. AFAIK, nobody ever stuck with it long enough to track down the real cause, though. This relatively recent comment in pythonrun.c's Py_Finalize() is probably relevant: /* The interpreter is still entirely intact at this point, and the * exit funcs may be relying on that. In particular, if some thread * or exit func is still waiting to do an import, the import machinery * expects Py_IsInitialized() to return true. So don't say the * interpreter is uninitialized until after the exit funcs have run. * Note that Threading.py uses an exit func to do a join on all the * threads created thru it, so this also protects pending imports in * the threads created via Threading. */ call_sys_exitfunc(); initialized = 0; I expect that a *foreign* thread (== one not created by threading.py) isn't going to be waited for by threading.py's exit func .join(), and if so won't be protected by this mechanism (that's a shutdown race we can't really do anything about -- we've got no handle on what foreign threads do). From gmcm@hypernet.com Fri Oct 5 20:20:33 2001 From: gmcm@hypernet.com (Gordon McMillan) Date: Fri, 5 Oct 2001 15:20:33 -0400 Subject: [Python-Dev] imputil and modulefinder replacements In-Reply-To: <20011005121931.B1801@cthulhu.mems-exchange.org> References: <3BBDDA83.B28556CB@lemburg.com> Message-ID: <3BBDCFC1.26732.68AF7945@localhost> [Greg Ward] > The only import hook I'm familiar with is Quixote, where we have > defined a Python dialect called PTL (Python Template Language, > used to embed HTML [or other text] in Python). Install Quixote's > import hook, and you can import .ptl files just like .py files. > It's very handy. Out of curiosity, does anyone know of any other > import hooks like this out there -- ie. import something that is > not strictly Python. (It sounds like most import hooks/hacks > deal with the location of the .py files to import. Quixote > doesn't touch that, but it adds the ability to import > not-quite-Python source files.) I know of import hooks like that, but in 1000+ emails from Installer users, no one has ever attempted to distribute an app that uses one. If the hook writes out a .pyc, then the hook is pretty much developer-only, anyway. > Note that my experience of this in Java was largely negative, > because Java doesn't have a standard way of putting .class files > in the filesystem (AFAIK) -- so everything has to be in a .jar > file, and those .jar files can be anywhere you please. Huh? Jar files don't date from day 1 - for .class files it works almost like Python. > So you > end up with a mile-long CLASSPATH that's very fragile and forever > needing fixing. As long as most Python modules are accessed the > ordinary way (files in a directory), then Python won't have a > problem. But if somebody makes a Python installation with > "stdlib.zip", "distutils.zip", "mxDateTime.zip", etc. etc., then > the poor users will be in the same boat as Java users. I think the only difference is that Python goes to considerable effort (and expense) to work out a sys.path before considering PYTHONPATH. - Gordon From gmcm@hypernet.com Fri Oct 5 20:20:32 2001 From: gmcm@hypernet.com (Gordon McMillan) Date: Fri, 5 Oct 2001 15:20:32 -0400 Subject: [Python-Dev] imputil and modulefinder replacements In-Reply-To: <3BBDD254.32DC4071@interet.com> Message-ID: <3BBDCFC0.18517.68AF780F@localhost> [James C. Ahlstrom] > My interest in imputil and ImportManager is to implement jar > files, and neither does this well. The problem is always how to > get the SpecialImporter.py imported first, and how to hack the > imports that the SpecialImporter.py may need to import. In > iu4.py we see the use of posix.stat to replace the os module, and > the import of only built-in modules sys, imp, and marshal. The > same is true of imputil.py. Here's my bootstrap code (archive_rt knows how to import from .pyz archives): import archive_rt, iu4 iu4._globalownertypes.insert(0, archive_rt.PYZOwner) iu4.ImportManager().install() Now a .pyz on sys.path will be searched. Nothing hard about doing this for a .zip. MacPython & Jython do it. > I admit I don't really understand package imports. But I feel > they are an unsolved problem, because package authors always seem > to write custom ones. Most of what I see is namespace trickery, not custom importers. > A first-class solution would have these features IMHO: > 1) Access to libraries in zip files is implemented in C (I > volunteer). 2) All Python library modules can be imported by C > code, > either from directories or zip files. With Thomas's changes to PyImport_ImportModuleEx, this is largely taken care of (it gets routed through __import__). So with the above bootstrap code, C code will import from my .pyz as well. > 3) Package import is not implemented in C, it is implemented > by a Python module which is part of the standard library. The rules aren't documented, but the essentials are pretty straightforward. I'd like to see them changed to draw a distinction between relative imports and absolute imports (since that's a constant source of gotcha's). However, there are import hacks out there that rely on undocumented (and probably accidental) features of implementation. > 4) > If an import of A.B is attempted, and a function > A.MyCustomImporter() exists, then that is used instead of the > default. In iu4, A is welcome to take over all imports below A. If A.__init__ does not do so, the standard mechanism is used. > 5) More or better ideas from you all. - Gordon From fdrake@acm.org Fri Oct 5 20:17:57 2001 From: fdrake@acm.org (Fred L. Drake, Jr.) Date: Fri, 5 Oct 2001 15:17:57 -0400 (EDT) Subject: [Python-Dev] os.spawnv[e] and related constants In-Reply-To: <15294.1645.63995.884978@beluga.mojam.com> References: <15294.1645.63995.884978@beluga.mojam.com> Message-ID: <15294.1893.967184.997647@cj42289-a.reston1.va.home.com> Skip Montanaro writes: > The spawnv and spawnve functions in the os module look useful, but the lib > manual documentation is rather sparse about their semantics, referring the > reader to the VC++ Runtime Library for more info. Perhaps this is poetic Are you looking at the development docs, or the 2.1.1 docs? http://python.sourceforge.net/devel-docs/lib/lib.html If you're looking at the devel docs and still find things are missing, please file a bug report with specific info about what you think is missing, and assign it to me. -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> PythonLabs at Zope Corporation From thomas.heller@ion-tof.com Fri Oct 5 20:34:14 2001 From: thomas.heller@ion-tof.com (Thomas Heller) Date: Fri, 5 Oct 2001 21:34:14 +0200 Subject: [Python-Dev] threading bug? References: <LNBBLJKPBEHFEDALKOLCOEIFLPAA.tim.one@home.com> Message-ID: <0b7e01c14dd4$b799a5b0$e000a8c0@thomasnotebook> From: "Tim Peters" <tim.one@home.com> > [Thomas Heller] > > Unhandled exception in thread: > > Traceback (most recent call last): > > File "c:\python21\lib\threading.py", line 393, in __bootstrap > > self.__stop() > > File "c:\python21\lib\threading.py", line 399, in __stop > > self.__block.notifyAll() > > File "c:\python21\lib\threading.py", line 235, in notifyAll > > self.notify(len(self.__waiters)) > > File "c:\python21\lib\threading.py", line 217, in notify > > me = currentThread() > > TypeError: object of type 'None' is not callable > > > > This traceback does not appear when I register an atexit > > function to stop the thread. > > We're missing basic info, like which OS and which Python release. More than > one shutdown race has been fixed over the decades <wink>. > > I've seen reports of this before, but under older Pythons, and/or in > conjunction with running GUIs where the problem went away if the GUI was > taken out of the equation. AFAIK, nobody ever stuck with it long enough to > track down the real cause, though. Here's the short info (I can provide everything to produce this, but not today). ActiveState's Python 2.1.1, running under Win2k SP1. Haven't tried in other Python versions. No gui involved. The thread basically calls os.stat, not sure if it also calls reload(). Later, Thomas From tim.one@home.com Fri Oct 5 20:42:48 2001 From: tim.one@home.com (Tim Peters) Date: Fri, 5 Oct 2001 15:42:48 -0400 Subject: [Python-Dev] os.spawnv[e] and related constants In-Reply-To: <15294.1645.63995.884978@beluga.mojam.com> Message-ID: <LNBBLJKPBEHFEDALKOLCEEIJLPAA.tim.one@home.com> [Skip Montanaro] > The spawnv and spawnve functions in the os module look useful, but the > lib manual documentation is rather sparse about their semantics, > referring the reader to the VC++ Runtime Library for more info. Gigabytes of MS docs (including all the VC docs) are available online for free: http://msdn.microsoft.com/library/ Since the MS docs are copyrighted, we can't just copy them into our docs. > ... > I can guess about P_WAIT and P_NOWAIT, but the > meanings of the others aren't obvious to a person without Windows > experience. They aren't obvious to anyone, and if you want to use the non-obvious ones, there are reams of MS docs you should understand first: <http://msdn.microsoft.com/library/default.asp?url=/library/en-us/vccore98/H TML/_crt__spawn.2c_._wspawn_functions.asp> How is Fred going to summarize all that (all of it is necessary in the end!), without running afoul of copyright? Alas, it can take a long time to learn how to find this stuff (the doc collection is massive). In general, the place to start for C library docs is: Visual Tools and Languages -> Visual Studio 6.0 -> Vlsual C and C++ 6.0 -> Product Documentation -> Visual C++ Programmer's Guide -> Run-Time Library Reference From tim.one@home.com Fri Oct 5 20:48:20 2001 From: tim.one@home.com (Tim Peters) Date: Fri, 5 Oct 2001 15:48:20 -0400 Subject: [Python-Dev] threading bug? In-Reply-To: <0b7e01c14dd4$b799a5b0$e000a8c0@thomasnotebook> Message-ID: <LNBBLJKPBEHFEDALKOLCAEIKLPAA.tim.one@home.com> [Thomas Heller] > Here's the short info (I can provide everything to produce this, > but not today). You overestimate my interest <wink>. > ActiveState's Python 2.1.1, running under Win2k SP1. > Haven't tried in other Python versions. > No gui involved. > The thread basically calls os.stat, not sure if it also calls reload(). If you can produce a reasonably small failing program, that doesn't involve foreign threads (as before, we've got no handle on those), attach it to a new bug report on SourceForge. If I can reproduce it in core Python, I'll find a way to fix it; else it's unique to ActiveState's distro, and they'll have to tackle it. From guido@python.org Fri Oct 5 20:50:23 2001 From: guido@python.org (Guido van Rossum) Date: Fri, 05 Oct 2001 15:50:23 -0400 Subject: [Python-Dev] threading bug? In-Reply-To: Your message of "Fri, 05 Oct 2001 15:48:20 EDT." <LNBBLJKPBEHFEDALKOLCAEIKLPAA.tim.one@home.com> References: <LNBBLJKPBEHFEDALKOLCAEIKLPAA.tim.one@home.com> Message-ID: <200110051950.PAA31207@cj20424-a.reston1.va.home.com> > If you can produce a reasonably small failing program, that doesn't involve > foreign threads (as before, we've got no handle on those), attach it to a > new bug report on SourceForge. If I can reproduce it in core Python, I'll > find a way to fix it; else it's unique to ActiveState's distro, and they'll > have to tackle it. I think he mentioned it was a daemon thread, and those are not stopped -- they are let run just like foreign threads. I still believe in my theory which I posted earlier today. --Guido van Rossum (home page: http://www.python.org/~guido/) From tim.one@home.com Fri Oct 5 20:58:44 2001 From: tim.one@home.com (Tim Peters) Date: Fri, 5 Oct 2001 15:58:44 -0400 Subject: [Python-Dev] threading bug? In-Reply-To: <200110051950.PAA31207@cj20424-a.reston1.va.home.com> Message-ID: <LNBBLJKPBEHFEDALKOLCOEIKLPAA.tim.one@home.com> [Guido] > I think he mentioned it was a daemon thread, He wasn't clear about that. > and those are not stopped -- they are let run just like foreign threads. I'm going to redefine what I mean by "foreign" then, to cover this too. > I still believe in my theory which I posted earlier today. So do I. From skip@pobox.com (Skip Montanaro) Fri Oct 5 21:37:59 2001 From: skip@pobox.com (Skip Montanaro) (Skip Montanaro) Date: Fri, 5 Oct 2001 15:37:59 -0500 Subject: [Python-Dev] os.spawnv[e] and related constants In-Reply-To: <15294.1893.967184.997647@cj42289-a.reston1.va.home.com> References: <15294.1645.63995.884978@beluga.mojam.com> <15294.1893.967184.997647@cj42289-a.reston1.va.home.com> Message-ID: <15294.6695.493437.770967@beluga.mojam.com> >> The spawnv and spawnve functions in the os module look useful, ... Fred> Are you looking at the development docs, or the 2.1.1 docs? I was looking at the 2.1.1 docs, but the development docs have the same problem w.r.t. the P_* constants. (The spawn* docs look much better.) I filed a bug report about the P_* documentation. Skip From skip@pobox.com (Skip Montanaro) Fri Oct 5 21:50:13 2001 From: skip@pobox.com (Skip Montanaro) (Skip Montanaro) Date: Fri, 5 Oct 2001 15:50:13 -0500 Subject: [Python-Dev] os.spawnv[e] and related constants In-Reply-To: <LNBBLJKPBEHFEDALKOLCEEIJLPAA.tim.one@home.com> References: <15294.1645.63995.884978@beluga.mojam.com> <LNBBLJKPBEHFEDALKOLCEEIJLPAA.tim.one@home.com> Message-ID: <15294.7429.716150.204682@beluga.mojam.com> >>>>> "Tim" == Tim Peters <tim.one@home.com> writes: Tim> [Skip Montanaro] >> The spawnv and spawnve functions in the os module look useful, but >> the lib manual documentation is rather sparse about their semantics, >> referring the reader to the VC++ Runtime Library for more info. Tim> Gigabytes of MS docs (including all the VC docs) are available Tim> online for free: Tim> http://msdn.microsoft.com/library/ Tim> Since the MS docs are copyrighted, we can't just copy them into our Tim> docs. ... Tim> <http://msdn.microsoft.com/library/default.asp?url=/library/en-us/vccore98/HTML/_crt__spawn.2c_._wspawn_functions.asp> Perhaps the Python docs could have a "See Also" that refers to this page? Tim> How is Fred going to summarize all that (all of it is necessary in Tim> the end!), without running afoul of copyright? It appears that in the development docs Fred has done a good job of summarizing the spawn functions' semantics. It's only the P_* constants whose definition is still left unclear. Copyright has the notion of "fair use". Since the information extracted would be so small w.r.t. the whole work from which it was extracted and it would not be used for commercial gain (at least, not directly), I would think that describing the five constants would fall under fair use guidelines. Tim> Alas, it can take a long time to learn how to find this stuff (the Tim> doc collection is massive). In general, the place to start for C Tim> library docs is: Thanks, but I'm not really interested in programming Windows, just knowing what the stuff imported from Windows means. ;-) Skip From guido@python.org Fri Oct 5 22:01:22 2001 From: guido@python.org (Guido van Rossum) Date: Fri, 05 Oct 2001 17:01:22 -0400 Subject: [Python-Dev] Need a way to test for 8-bit-or-unicode-string In-Reply-To: Your message of "Fri, 05 Oct 2001 20:21:29 +0200." <3BBDFA29.A5A4DCBA@lemburg.com> References: <200110051436.KAA24983@cj20424-a.reston1.va.home.com> <3BBDD79D.C26D1646@lemburg.com> <200110051557.LAA25625@cj20424-a.reston1.va.home.com> <15293.57386.311536.729065@anthem.wooz.org> <3BBDE670.52DFB7F0@lemburg.com> <15293.59431.1516.201622@anthem.wooz.org> <3BBDFA29.A5A4DCBA@lemburg.com> Message-ID: <200110052101.RAA02367@cj20424-a.reston1.va.home.com> Now that the introduction of a class hierarchy is dead, I'd like to move forward with the isinstance(x, (A, B, C...)) patch. Followup question: should I also implement issubclass(X, (A, B, C...))? It's consistent to do so, but it's more work (issubclass is a bit murkier due to the "abstract subclass" support), and I'm not sure that it has the same benefits... --Guido van Rossum (home page: http://www.python.org/~guido/) From tim.one@home.com Fri Oct 5 23:33:47 2001 From: tim.one@home.com (Tim Peters) Date: Fri, 5 Oct 2001 18:33:47 -0400 Subject: [Python-Dev] Painful death in debug build Message-ID: <LNBBLJKPBEHFEDALKOLCKEJGLPAA.tim.one@home.com> test_descr.py crashes under the debug build now. I've got too many things piled on my stack to look into it right now, so it would be good if someone could try to feel debug pain on Linux too. The top of the call stack is _free_dbg_lk(void * 0x009e5fd0, int 1) line 1066 + 60 bytes _free_dbg(void * 0x009e5fd0, int 1) line 1001 + 13 bytes free(void * 0x009e5fd0) line 956 + 11 bytes _PyObject_GC_Del(_object * 0x009e5fdc) line 873 + 10 bytes long_dealloc(_object * 0x009e5fdc) line 1230 + 16 bytes subtype_dealloc(_object * 0x009e5fdc) line 301 + 7 bytes _Py_Dealloc(_object * 0x009e5fdc) line 1738 + 7 bytes eval_frame(_frame * 0x00803d7c) line 845 + 59 bytes PyEval_EvalCodeEx(...) at this point, so first guess is that it has to do with adding new-style instances to GC. The MS debug library free() is complaining because it detects damage (corruption) in the pad bytes debug malloc() writes *between* the blocks it hands out. So it's unlikely that it points to the direct cause; it's just detecting that user code, somewhere, sometime, wrote over memory it had no business writing to. From martin@loewis.home.cs.tu-berlin.de Sat Oct 6 00:54:03 2001 From: martin@loewis.home.cs.tu-berlin.de (Martin v. Loewis) Date: Sat, 6 Oct 2001 01:54:03 +0200 Subject: [Python-Dev] Painful death in debug build Message-ID: <200110052354.f95Ns3007683@mira.informatik.hu-berlin.de> > test_descr.py crashes under the debug build now. I've got too many > things piled on my stack to look into it right now, so it would be > good if someone could try to feel debug pain on Linux too. Configuring Python with --with-pydebug, I cannot get test_descr to crash on Linux, using the current CVS. Regards, Martin From dariob@millic.com.ar Sat Oct 6 01:19:01 2001 From: dariob@millic.com.ar (dariob@millic.com.ar) Date: Fri, 05 Oct 2001 21:19:01 -0300 Subject: [Python-Dev] VCD Y MP3. Message-ID: <04fc632160006a1MAIL1@smtp1.millic.com.ar> This is a Multipart MIME message. ------=_NextPart_000_001__2771585_76741,02 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: 7bit ------=_NextPart_000_001__2771585_76741,02 Content-Type: text/html; charset=iso-8859-1 Content-Transfer-Encoding: base64 PGh0bWw+DQoNCjxoZWFkPg0KPG1ldGEgaHR0cC1lcXVpdj0iQ29udGVudC1MYW5ndWFnZSIg Y29udGVudD0iZXMiPg0KPG1ldGEgbmFtZT0iR0VORVJBVE9SIiBjb250ZW50PSJNaWNyb3Nv ZnQgRnJvbnRQYWdlIDUuMCI+DQo8bWV0YSBuYW1lPSJQcm9nSWQiIGNvbnRlbnQ9IkZyb250 UGFnZS5FZGl0b3IuRG9jdW1lbnQiPg0KPG1ldGEgaHR0cC1lcXVpdj0iQ29udGVudC1UeXBl IiBjb250ZW50PSJ0ZXh0L2h0bWw7IGNoYXJzZXQ9d2luZG93cy0xMjUyIj4NCjx0aXRsZT5Q YWdpbmEgbnVldmEgMTwvdGl0bGU+DQo8L2hlYWQ+DQoNCjxib2R5Pg0KDQo8dGFibGUgaGVp Z2h0PSI4MSIgY2VsbFNwYWNpbmc9IjAiIGNlbGxQYWRkaW5nPSIwIiB3aWR0aD0iMTAwJSIg YmdDb2xvcj0iIzAwMDAwMCIgYm9yZGVyPSIwIj4NCiAgPGZvbnQgZmFjZT0iQXJpYWwsIEhl bHZldGljYSwgc2Fucy1zZXJpZiIgY29sb3I9IiMwMDgwZmYiIHNpemU9IjQiPjxiPg0KICA8 Y2FwdGlvbj48Zm9udCBmYWNlPSJUYWhvbWEiIGNvbG9yPSIjZmYwMDAwIiBzaXplPSI2Ij48 c3Ryb25nPqGhoaEgTk8gVEUgUE9ERVMgDQogIFBFUkRFUiBFU1RBIE9QT1JUVU5JREFEISEh ITwvc3Ryb25nPjwvZm9udD4gPHN0cm9uZz4NCiAgPGZvbnQgZmFjZT0iVGFob21hIiBjb2xv cj0iIzAwMDBmZiIgc2l6ZT0iNSI+PGZvbnQgc2l6ZT0iNSIgZmFjZT0iVGFob21hIj4NCiAg RVNDVUNIQSA8L2ZvbnQ+PC9mb250Pjwvc3Ryb25nPjxmb250IHNpemU9IisyIiBjb2xvcj0i IzAwMDBmZiIgZmFjZT0iVGFob21hIj4NCiAgTdpTSUNBIE1QMyBFTiBUVSBBVVRPLCBDQVNB LCBDQU1JTkFORE8gTyBFTiBET05ERSBQUkVGSUVSQVMsPC9mb250PjxzdHJvbmc+PGZvbnQg ZmFjZT0iVGFob21hIiBjb2xvcj0iIzAwMDBmZiIgc2l6ZT0iNSI+PGZvbnQgc2l6ZT0iNSIg ZmFjZT0iVGFob21hIj4NCiAgWSBNSVJBIExBUyBNRUpPUkVTIDwvZm9udD48Zm9udCBzaXpl PSI1IiBmYWNlPSJUYWhvbWEiPg0KICBQRUzNQ1VMQVMgRU4gRUwgVFYgREUgVFUgQ0FTQS48 L2ZvbnQ+PC9mb250Pjwvc3Ryb25nPg0KICA8Zm9udCBmYWNlPSJUYWhvbWEiIGNvbG9yPSIj MDAwMGZmIiBzaXplPSI1Ij48c3Ryb25nPjx1PlPaUEVSIE9GRVJUQSBFTiBMQSANCiAgUFVF UlRBIERFIFRVIENBU0E8L3U+PC9zdHJvbmc+PC9mb250PjwvYj48L2ZvbnQ+DQogIDxwIGFs aWduPSJjZW50ZXIiPg0KICA8Zm9udCBmYWNlPSJBcmlhbCwgSGVsdmV0aWNhLCBzYW5zLXNl cmlmIiBjb2xvcj0iIzAwODBmZiIgc2l6ZT0iNCI+PGI+TG8gTWVqb3IgDQogIGRlbCBNZXJj YWRvIGEgUHJlY2lvIGRlIExPQ09TPC9iPjwvZm9udD48L3A+DQogIDwvY2FwdGlvbj4NCiAg PHRyPg0KICAgIDx0ZCBoZWlnaHQ9IjgxIj4NCiAgICA8dGFibGUgY2VsbFNwYWNpbmc9IjAi IGNlbGxQYWRkaW5nPSI2IiB3aWR0aD0iMTAwJSIgYm9yZGVyPSIwIiBoZWlnaHQ9IjcxIj4N CiAgICAgIDx0cj4NCiAgICAgICAgPHRkIHdpZHRoPSI1MyUiIGhlaWdodD0iNTkiPg0KICAg ICAgICA8dGFibGUgY2VsbFNwYWNpbmc9IjAiIGNlbGxQYWRkaW5nPSIzIiB3aWR0aD0iMTAw JSIgYmdDb2xvcj0iIzAwODAwMCIgYm9yZGVyPSIwIj4NCiAgICAgICAgICA8dHI+DQogICAg ICAgICAgICA8dGQgd2lkdGg9IjEwMCUiPg0KICAgICAgICAgICAgPHAgc3R5bGU9IndvcmQt c3BhY2luZzogMHB4OyBtYXJnaW4tbGVmdDogMHB4OyBtYXJnaW4tcmlnaHQ6IDBweDsgbWFy Z2luLXRvcDogMHB4OyBtYXJnaW4tYm90dG9tOiA2cHgiIGFsaWduPSJjZW50ZXIiPg0KICAg ICAgICAgICAgPGZvbnQgZmFjZT0iQXJpYWwgQmxhY2siIGNvbG9yPSIjZmZmZjAwIj5PZmVy dGEgbGltaXRhZGEsIGhhc3RhIA0KICAgICAgICAgICAgYWdvdGFyIFN0b2NrLjwvZm9udD48 L3A+DQogICAgICAgICAgICA8cCBzdHlsZT0id29yZC1zcGFjaW5nOiAwcHg7IG1hcmdpbi1s ZWZ0OiAwcHg7IG1hcmdpbi1yaWdodDogMHB4OyBtYXJnaW4tdG9wOiAwcHg7IG1hcmdpbi1i b3R0b206IDZweCIgYWxpZ249ImNlbnRlciI+DQogICAgICAgICAgICA8Yj48Zm9udCBmYWNl PSJBcmlhbCIgY29sb3I9IiNmZmZmMDAiIHNpemU9IjIiPk11Y2hvcyB5YSBsbyB0aWVuZW4s IA0KICAgICAgICAgICAgZXN0YSBvcG9ydHVuaWRhZCBubyBzZSByZXBldGly4SB5IGxhcyB1 bmlkYWRlcyBzZSBlc3RhbiBhY2FiYW5kbywgbm8gDQogICAgICAgICAgICBzZSB2ZW5kZSBw b3Igb3RybyBtZWRpbywgaGF6IHR1IG9mZXJ0YSBwdWVzIG1h8WFuYSBwdWVkZSBzZXIgdGFy ZGUuPC9mb250PjwvYj48L3RkPg0KICAgICAgICAgIDwvdHI+DQogICAgICAgIDwvdGFibGU+ DQogICAgICAgIDwvdGQ+DQogICAgICA8L3RyPg0KICAgIDwvdGFibGU+DQogICAgPC90ZD4N CiAgPC90cj4NCjwvdGFibGU+DQo8dGFibGUgaGVpZ2h0PSIxNTQiIGNlbGxTcGFjaW5nPSIw IiBjZWxsUGFkZGluZz0iMCIgd2lkdGg9IjU4NCIgYm9yZGVyPSIwIj4NCiAgPHRyPg0KICAg IDx0ZCB2QWxpZ249InRvcCIgd2lkdGg9IjI3MSIgaGVpZ2h0PSIxNTQiPg0KICAgIDx0YWJs ZSBoZWlnaHQ9IjQxNSIgY2VsbFNwYWNpbmc9IjAiIGNlbGxQYWRkaW5nPSIwIiB3aWR0aD0i MjA3IiBib3JkZXI9IjAiPg0KICAgICAgPHRyPg0KICAgICAgICA8dGQgd2lkdGg9IjE5NyIg aGVpZ2h0PSI0MTUiPg0KICAgICAgICA8dGFibGUgYm9yZGVyQ29sb3I9IiMwMDgwZmYiIGhl aWdodD0iMzQxIiBjZWxsU3BhY2luZz0iMCIgY2VsbFBhZGRpbmc9IjAiIHdpZHRoPSIxMDAl IiBib3JkZXI9IjIiPg0KICAgICAgICAgIDx0cj4NCiAgICAgICAgICAgIDx0ZCB3aWR0aD0i MTAwJSIgaGVpZ2h0PSIzMSI+DQogICAgICAgICAgICA8dGFibGUgY2VsbFNwYWNpbmc9IjAi IGNlbGxQYWRkaW5nPSI1IiB3aWR0aD0iMTAwJSIgYmdDb2xvcj0iIzAwODBmZiIgYm9yZGVy PSIwIj4NCiAgICAgICAgICAgICAgPHRyPg0KICAgICAgICAgICAgICAgIDx0ZCB3aWR0aD0i MTAwJSI+PGZvbnQgZmFjZT0iVGFob21hIiBjb2xvcj0iI2ZmZmZmZiIgc2l6ZT0iMyI+DQog ICAgICAgICAgICAgICAgPGI+Q2FyYWN0ZXLtc3RpY2FzPC9iPjwvZm9udD48L3RkPg0KICAg ICAgICAgICAgICA8L3RyPg0KICAgICAgICAgICAgPC90YWJsZT4NCiAgICAgICAgICAgIDwv dGQ+DQogICAgICAgICAgPC90cj4NCiAgICAgICAgICA8dHI+DQogICAgICAgICAgICA8dGQg d2lkdGg9IjEwMCUiIGhlaWdodD0iMzA2Ij4NCiAgICAgICAgICAgIDx0YWJsZSBoZWlnaHQ9 IjMwMiIgY2VsbFNwYWNpbmc9IjAiIGNlbGxQYWRkaW5nPSI1IiB3aWR0aD0iMTAwJSIgYm9y ZGVyPSIwIj4NCiAgICAgICAgICAgICAgPHRyPg0KICAgICAgICAgICAgICAgIDx0ZCB3aWR0 aD0iMTAwJSIgaGVpZ2h0PSIyOTYiPg0KICAgICAgICAgICAgICAgIDx0YWJsZSBjZWxsU3Bh Y2luZz0iMCIgY2VsbFBhZGRpbmc9IjAiIHdpZHRoPSIxMDAlIiBib3JkZXI9IjAiPg0KICAg ICAgICAgICAgICAgICAgPHRyPg0KICAgICAgICAgICAgICAgICAgICA8dGQgd2lkdGg9IjEw MCUiPg0KICAgICAgICAgICAgICAgICAgICA8cCBzdHlsZT0id29yZC1zcGFjaW5nOiAwcHg7 IG1hcmdpbi1sZWZ0OiAwcHg7IG1hcmdpbi1yaWdodDogMHB4OyBtYXJnaW4tdG9wOiAwcHg7 IG1hcmdpbi1ib3R0b206IDVweCI+DQogICAgICAgICAgICAgICAgICAgIDxmb250IGZhY2U9 IlRhaG9tYSIgY29sb3I9IiMwMDgwZmYiIHNpemU9IjIiPjxiPlVuaWNvIGVuIA0KICAgICAg ICAgICAgICAgICAgICBBcmdlbnRpbmEgcXVlIGxlZSBjZCBjb24gTVAzIHkgYWRlbWFzIGxl ZSBWQ0QgY29uIHNhbGlkYSBkZSANCiAgICAgICAgICAgICAgICAgICAgdmlkZW8geSBhdWRp byBvIHNlYSBxdWUmbmJzcDtzZSB2ZW4gZW4gZWwgdGVsZXZpc29yLCBlcyBsbyB1bHRpbW8g DQogICAgICAgICAgICAgICAgICAgIGVuIHRlY25vbG9naWEgcGFyYSBtcDMgeSB2Y2Q8L2I+ PC9mb250PjwvcD4NCiAgICAgICAgICAgICAgICAgICAgPHAgc3R5bGU9IndvcmQtc3BhY2lu ZzogMHB4OyBtYXJnaW4tbGVmdDogMHB4OyBtYXJnaW4tcmlnaHQ6IDBweDsgbWFyZ2luLXRv cDogMHB4OyBtYXJnaW4tYm90dG9tOiA1cHgiPg0KICAgICAgICAgICAgICAgICAgICA8Zm9u dCBmYWNlPSJUYWhvbWEiIGNvbG9yPSIjMDA4MGZmIiBzaXplPSIyIj48Yj5FbCANCiAgICAg ICAgICAgICAgICAgICAgcmVwcm9kdWN0b3IgZGUgTVAzIHkgVkNEIGluY2x1eWU6PC9iPjwv Zm9udD48L3A+DQogICAgICAgICAgICAgICAgICAgIDxwIHN0eWxlPSJ3b3JkLXNwYWNpbmc6 IDBweDsgbWFyZ2luLWxlZnQ6IDBweDsgbWFyZ2luLXJpZ2h0OiAwcHg7IG1hcmdpbi10b3A6 IDBweDsgbWFyZ2luLWJvdHRvbTogNXB4Ij4NCiAgICAgICAgICAgICAgICAgICAgPGZvbnQg ZmFjZT0iVGFob21hIiBjb2xvcj0iIzAwODBmZiIgc2l6ZT0iMiI+PGI+LSBDb250cm9sIA0K ICAgICAgICAgICAgICAgICAgICByZW1vdG8gaW5hbGFtYnJpY288L2I+PC9mb250PjwvcD4N CiAgICAgICAgICAgICAgICAgICAgPHAgc3R5bGU9IndvcmQtc3BhY2luZzogMHB4OyBtYXJn aW4tbGVmdDogMHB4OyBtYXJnaW4tcmlnaHQ6IDBweDsgbWFyZ2luLXRvcDogMHB4OyBtYXJn aW4tYm90dG9tOiA1cHgiPg0KICAgICAgICAgICAgICAgICAgICA8Zm9udCBmYWNlPSJUYWhv bWEiIGNvbG9yPSIjMDA4MGZmIiBzaXplPSIyIj48Yj4tIEJhdGVyaWFzIA0KICAgICAgICAg ICAgICAgICAgICAoUGlsYXMpIFJlY2FyZ2FibGVzIExpdGl1bTwvYj48L2ZvbnQ+PC9wPg0K ICAgICAgICAgICAgICAgICAgICA8cCBzdHlsZT0id29yZC1zcGFjaW5nOiAwcHg7IG1hcmdp bi1sZWZ0OiAwcHg7IG1hcmdpbi1yaWdodDogMHB4OyBtYXJnaW4tdG9wOiAwcHg7IG1hcmdp bi1ib3R0b206IDVweCI+DQogICAgICAgICAgICAgICAgICAgIDxmb250IGZhY2U9IlRhaG9t YSIgY29sb3I9IiMwMDgwZmYiIHNpemU9IjIiPjxiPi0gQ2FibGVzIGRlIA0KICAgICAgICAg ICAgICAgICAgICBBdWRpbyB5IGRlIFZpZGVvPC9iPjwvZm9udD48L3A+DQogICAgICAgICAg ICAgICAgICAgIDxwIHN0eWxlPSJ3b3JkLXNwYWNpbmc6IDBweDsgbWFyZ2luLWxlZnQ6IDBw eDsgbWFyZ2luLXJpZ2h0OiAwcHg7IG1hcmdpbi10b3A6IDBweDsgbWFyZ2luLWJvdHRvbTog NXB4Ij4NCiAgICAgICAgICAgICAgICAgICAgPGZvbnQgZmFjZT0iVGFob21hIiBjb2xvcj0i IzAwODBmZiIgc2l6ZT0iMiI+PGI+LSBBdWRpZm9ubzwvYj48L2ZvbnQ+PC9wPg0KICAgICAg ICAgICAgICAgICAgICA8cCBzdHlsZT0id29yZC1zcGFjaW5nOiAwcHg7IG1hcmdpbi1sZWZ0 OiAwcHg7IG1hcmdpbi1yaWdodDogMHB4OyBtYXJnaW4tdG9wOiAwcHg7IG1hcmdpbi1ib3R0 b206IDVweCI+DQogICAgICAgICAgICAgICAgICAgIDxmb250IGZhY2U9IlRhaG9tYSIgY29s b3I9IiMwMDgwZmYiIHNpemU9IjIiPjxiPi0gTWFudWFsIGRlIA0KICAgICAgICAgICAgICAg ICAgICBpbnN0cnVjY2lvbjwvYj48L2ZvbnQ+PC9wPg0KICAgICAgICAgICAgICAgICAgICA8 cCBzdHlsZT0id29yZC1zcGFjaW5nOiAwcHg7IG1hcmdpbi1sZWZ0OiAwcHg7IG1hcmdpbi1y aWdodDogMHB4OyBtYXJnaW4tdG9wOiAwcHg7IG1hcmdpbi1ib3R0b206IDVweCI+DQogICAg ICAgICAgICAgICAgICAgIDxmb250IGZhY2U9IlRhaG9tYSIgY29sb3I9IiMwMDgwZmYiIHNp emU9IjIiPjxiPi0gDQogICAgICAgICAgICAgICAgICAgIFRyYW5zZm9ybWFkb3I8L2I+PC9m b250PjwvcD4NCiAgICAgICAgICAgICAgICAgICAgPHAgc3R5bGU9IndvcmQtc3BhY2luZzog MHB4OyBtYXJnaW4tbGVmdDogMHB4OyBtYXJnaW4tcmlnaHQ6IDBweDsgbWFyZ2luLXRvcDog MHB4OyBtYXJnaW4tYm90dG9tOiA1cHgiPg0KICAgICAgICAgICAgICAgICAgICA8Zm9udCBm YWNlPSJUYWhvbWEiIGNvbG9yPSIjMDA4MGZmIiBzaXplPSIyIj48Yj4tIENhcmdhZG9yPC9i PjwvZm9udD48L3RkPg0KICAgICAgICAgICAgICAgICAgPC90cj4NCiAgICAgICAgICAgICAg ICA8L3RhYmxlPg0KICAgICAgICAgICAgICAgIDwvdGQ+DQogICAgICAgICAgICAgIDwvdHI+ DQogICAgICAgICAgICA8L3RhYmxlPg0KICAgICAgICAgICAgPC90ZD4NCiAgICAgICAgICA8 L3RyPg0KICAgICAgICA8L3RhYmxlPg0KICAgICAgICA8L3RkPg0KICAgICAgPC90cj4NCiAg ICA8L3RhYmxlPg0KICAgIDwvdGQ+DQogICAgPHRkIHdpZHRoPSIzMjciIGhlaWdodD0iMTU0 Ij4NCiAgICA8dGFibGUgY2VsbFNwYWNpbmc9IjAiIGNlbGxQYWRkaW5nPSIwIiB3aWR0aD0i MTAwJSIgYm9yZGVyPSIwIj4NCiAgICAgIDx0cj4NCiAgICAgICAgPHRkIHdpZHRoPSIxMDAl Ij4NCiAgICAgICAgPHRhYmxlIGNlbGxTcGFjaW5nPSIwIiBjZWxsUGFkZGluZz0iMCIgd2lk dGg9IjEwMCUiIGJvcmRlcj0iMCI+DQogICAgICAgICAgPHRyPg0KICAgICAgICAgICAgPHRk IHdpZHRoPSI3MiUiPg0KICAgICAgICAgICAgPGltZyBoZWlnaHQ9IjIxMiIgc3JjPSJodHRw Oi8vaW1hZ2Vob3N0LmF1Y3Rpb253YXRjaC5jb20vcHJldmlldy9jYy9jY29ycmVhL21wdHJl cDUuanBnIiB3aWR0aD0iMjIyIiBib3JkZXI9IjAiPjwvdGQ+DQogICAgICAgICAgICA8dGQg d2lkdGg9IjI4JSI+DQogICAgICAgICAgICA8dGFibGUgY2VsbFNwYWNpbmc9IjAiIGNlbGxQ YWRkaW5nPSIwIiB3aWR0aD0iMTAwJSIgYm9yZGVyPSIwIj4NCiAgICAgICAgICAgICAgPHRy Pg0KICAgICAgICAgICAgICAgIDx0ZCB3aWR0aD0iMTAwJSI+DQogICAgICAgICAgICAgICAg PGltZyBzcmM9Imh0dHA6Ly9pbWFnZWhvc3QuYXVjdGlvbndhdGNoLmNvbS9wcmV2aWV3L2Nj L2Njb3JyZWEvbXB0cmVwMS5qcGciIHdpZHRoPSIxMjYiIGJvcmRlcj0iMCI+PC90ZD4NCiAg ICAgICAgICAgICAgPC90cj4NCiAgICAgICAgICAgICAgPHRyPg0KICAgICAgICAgICAgICAg IDx0ZCB3aWR0aD0iMTAwJSI+DQogICAgICAgICAgICAgICAgPGltZyBzcmM9Imh0dHA6Ly9p bWFnZWhvc3QuYXVjdGlvbndhdGNoLmNvbS9wcmV2aWV3L2NjL2Njb3JyZWEvbXB0cmVwMi5q cGciIHdpZHRoPSIxMjYiIGJvcmRlcj0iMCI+PC90ZD4NCiAgICAgICAgICAgICAgPC90cj4N CiAgICAgICAgICAgICAgPHRyPg0KICAgICAgICAgICAgICAgIDx0ZCB3aWR0aD0iMTAwJSI+ DQogICAgICAgICAgICAgICAgPGltZyBzcmM9Imh0dHA6Ly9pbWFnZWhvc3QuYXVjdGlvbndh dGNoLmNvbS9wcmV2aWV3L2NjL2Njb3JyZWEvbXB0cmVwNy5qcGciIHdpZHRoPSIxMjYiIGJv cmRlcj0iMCI+PC90ZD4NCiAgICAgICAgICAgICAgPC90cj4NCiAgICAgICAgICAgICAgPHRy Pg0KICAgICAgICAgICAgICAgIDx0ZCB3aWR0aD0iMTAwJSI+DQogICAgICAgICAgICAgICAg PGltZyBzcmM9Imh0dHA6Ly9pbWFnZWhvc3QuYXVjdGlvbndhdGNoLmNvbS9wcmV2aWV3L2Nj L2Njb3JyZWEvbXB0cmVwOC5qcGciIHdpZHRoPSIxMjYiIGJvcmRlcj0iMCI+PC90ZD4NCiAg ICAgICAgICAgICAgPC90cj4NCiAgICAgICAgICAgIDwvdGFibGU+DQogICAgICAgICAgICA8 L3RkPg0KICAgICAgICAgIDwvdHI+DQogICAgICAgIDwvdGFibGU+DQogICAgICAgIDwvdGQ+ DQogICAgICA8L3RyPg0KICAgICAgPHRyPg0KICAgICAgICA8dGQgd2lkdGg9IjEwMCUiPg0K ICAgICAgICA8dGFibGUgY2VsbFNwYWNpbmc9IjAiIGNlbGxQYWRkaW5nPSIwIiB3aWR0aD0i MTAwJSIgYm9yZGVyPSIwIj4NCiAgICAgICAgICA8dHI+DQogICAgICAgICAgICA8dGQgd2lk dGg9IjUwJSI+DQogICAgICAgICAgICA8aW1nIGhlaWdodD0iMTMyIiBzcmM9Imh0dHA6Ly9p bWFnZWhvc3QuYXVjdGlvbndhdGNoLmNvbS9wcmV2aWV3L2NjL2Njb3JyZWEvbXB0cmVwMy5q cGciIHdpZHRoPSIxNjMiIGJvcmRlcj0iMCI+PC90ZD4NCiAgICAgICAgICAgIDx0ZCB3aWR0 aD0iNTAlIj4NCiAgICAgICAgICAgIDxpbWcgaGVpZ2h0PSIxMzMiIHNyYz0iaHR0cDovL2lt YWdlaG9zdC5hdWN0aW9ud2F0Y2guY29tL3ByZXZpZXcvY2MvY2NvcnJlYS9tcHRyZXA2Lmpw ZyIgd2lkdGg9IjE2OCIgYm9yZGVyPSIwIj48L3RkPg0KICAgICAgICAgIDwvdHI+DQogICAg ICAgIDwvdGFibGU+DQogICAgICAgIDwvdGQ+DQogICAgICA8L3RyPg0KICAgIDwvdGFibGU+ DQogICAgPC90ZD4NCiAgPC90cj4NCjwvdGFibGU+DQo8dGFibGUgaGVpZ2h0PSI0MCIgY2Vs bFNwYWNpbmc9IjAiIGNlbGxQYWRkaW5nPSIwIiB3aWR0aD0iNTg0IiBib3JkZXI9IjAiPg0K ICA8dHI+DQogICAgPHRkIGhlaWdodD0iNDAiPg0KICAgIDx0YWJsZSBjZWxsU3BhY2luZz0i MCIgY2VsbFBhZGRpbmc9IjAiIHdpZHRoPSIxMDAlIiBib3JkZXI9IjAiPg0KICAgICAgPHRy Pg0KICAgICAgICA8dGQgd2lkdGg9IjEwMCUiPg0KICAgICAgICA8dGFibGUgY2VsbFNwYWNp bmc9IjAiIGNlbGxQYWRkaW5nPSI1IiB3aWR0aD0iMTAwJSIgYmdDb2xvcj0iIzAwODBmZiIg Ym9yZGVyPSIwIj4NCiAgICAgICAgICA8dHI+DQogICAgICAgICAgICA8dGQgd2lkdGg9IjEw MCUiPjxiPjxmb250IGZhY2U9IlRhaG9tYSIgY29sb3I9IiNmZmZmZmYiIHNpemU9IjMiPg0K ICAgICAgICAgICAgRXNwZWNpZmljYWNpb25lcyB0ZWNuaWNhczwvZm9udD48L2I+PC90ZD4N CiAgICAgICAgICA8L3RyPg0KICAgICAgICA8L3RhYmxlPg0KICAgICAgICA8L3RkPg0KICAg ICAgPC90cj4NCiAgICAgIDx0cj4NCiAgICAgICAgPHRkIHdpZHRoPSIxMDAlIj4NCiAgICAg ICAgPHRhYmxlIGNlbGxTcGFjaW5nPSIwIiBjZWxsUGFkZGluZz0iNSIgd2lkdGg9IjEwMCUi IGJvcmRlcj0iMCI+DQogICAgICAgICAgPHRyPg0KICAgICAgICAgICAgPHRkIHdpZHRoPSI1 MCUiPjxiPjxmb250IGZhY2U9IlRhaG9tYSIgY29sb3I9IiMwMDgwZmYiIHNpemU9IjIiPg0K ICAgICAgICAgICAgU3Vtc3VuZyBMZW5zPGJyPg0KICAgICAgICAgICAgU29ueSBTZXJ2byBz eXN0ZW08YnI+DQogICAgICAgICAgICBTdXBwb3J0IFJlZ3VsYXIgQXVkaW8sIE1QMyxWQ0Qs RFZDRCw4MCBtaW4gQ0QtUixDRC1SVyBhbmQgOCBjbSBDRC1SIA0KICAgICAgICAgICAgZGlz Yzxicj4NCiAgICAgICAgICAgIGNvbXBhdGlibGUgVkNEIDIuMCwzLjAgYW5kIERWQ0QgZGlz Yzxicj4NCiAgICAgICAgICAgIDYgViBwb3dlciBvdXRwdXQ8YnI+DQogICAgICAgICAgICBi dWlsZC1pbiByZWNoYXJnZWFibGUgbGl0aGl1bSBiYXR0ZXJ5PGJyPg0KICAgICAgICAgICAg Q29uc3VtcHRpb24gb2YgYmF0dGVyeSAoc2kgZXMgcXVlIG5vIGxhIG9yaWdpbmFsKSAyLjUg LSAzIGhvdXJzIGZvciANCiAgICAgICAgICAgIE1QMyBkaXNjPGJyPg0KICAgICAgICAgICAg MS41IC0gMiBob3VycyBmb3IgVkNEIGRpc2M8YnI+DQombmJzcDs8L2ZvbnQ+PC9iPjwvdGQ+ DQogICAgICAgICAgICA8dGQgd2lkdGg9IjUwJSI+PGI+PGZvbnQgZmFjZT0iVGFob21hIiBj b2xvcj0iIzAwODBmZiIgc2l6ZT0iMiI+DQogICAgICAgICAgICBBY2Nlc3Nvcmllczo8YnI+ DQogICAgICAgICAgICBQb3dlciBzdXBwbHkoMTAwViAtIDI0MFYpPGJyPg0KICAgICAgICAg ICAgSW5zdHJ1Y3Rpb24gbWFudWFsPGJyPg0KICAgICAgICAgICAgRWFycGhvbmU8YnI+DQog ICAgICAgICAgICBSZWNoYXJnZWFibGUgTGl0aGl1bSBiYXR0ZXJpZXM8YnI+DQogICAgICAg ICAgICBBL1YgQ2FibGU8YnI+DQogICAgICAgICAgICBSZW1vdGUgY29udHJvbDxicj4NCiAg ICAgICAgICAgIEVzdGFzIHNvbiBsYXMgbm9ybWFzIGEgbGFzIHF1ZSBzZSBzb21ldGllcm9u IHkgcGFzYXJvbiBlc3RvIGF2YWxhIGxhIA0KICAgICAgICAgICAgY2FsaWRhZDxicj4NCiAg ICAgICAgICAgIGFwcHJvdmFsOiBGQ0MsIFVMLCBGREEsIENFLCBDQ0VFPC9mb250PjwvYj48 L3RkPg0KICAgICAgICAgIDwvdHI+DQogICAgICAgIDwvdGFibGU+DQogICAgICAgIDwvdGQ+ DQogICAgICA8L3RyPg0KICAgIDwvdGFibGU+DQogICAgPGRpdiBhbGlnbj0iY2VudGVyIj4N CiAgICAgIDxmb250IGZhY2U9IlRhaG9tYSIgc2l6ZT0iNSI+PHN0cm9uZz5Db24gbGEgY29t cHJhIGRlbCBT+nBlciBEaXNjbWFuIHRlIA0KICAgICAgbGxldmFzIGRlIHJlZ2FsbyA8L3N0 cm9uZz48L2ZvbnQ+DQogICAgPC9kaXY+DQogICAgPGRpdiBhbGlnbj0iY2VudGVyIj4NCiAg ICAgIDxmb250IGZhY2U9IlRhaG9tYSIgc2l6ZT0iNSI+PHN0cm9uZz51bmEgcOlsaWN1bGEg eSB1biBDRCBkZSBtcDMgYSANCiAgICAgIGVsZWNjafNuLjwvc3Ryb25nPjwvZm9udD48L2Rp dj4NCiAgICA8ZGl2IGFsaWduPSJjZW50ZXIiPg0KICAgICAgPGRpdiBhbGlnbj0iY2VudGVy Ij4NCiAgICAgICAgPGZvbnQgZmFjZT0iVGFob21hIiBjb2xvcj0iIzgwMDA4MCIgc2l6ZT0i NSI+PHN0cm9uZz4oTk8gVEUgUE9ERVMgUFJJVkFSIA0KICAgICAgICBERSBMQVMgTUVKT1JF UyBDT1NBUyk8L3N0cm9uZz48L2ZvbnQ+PC9kaXY+DQogICAgPC9kaXY+DQogICAgPC90ZD4N CiAgPC90cj4NCjwvdGFibGU+DQo8cD48Zm9udCBzaXplPSIrMiIgY29sb3I9IiM4MDAwODAi IGZhY2U9IlRhaG9tYSI+Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5i c3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5i c3A7Jm5ic3A7Jm5ic3A7IA0KRU5WSU9TIEEgVE9ETyBFTCBQQUlTPC9mb250PjwvcD4NCjxw Pjxmb250IHNpemU9IisyIiBjb2xvcj0iIzgwMDA4MCIgZmFjZT0iVGFob21hIj5QQVJBIE1B UyANCklORk9STUFDSU9OIFNPTE8gRU5WSUEgVU4gTUFJTCBBPC9mb250PjxzdHJvbmc+PGZv bnQgZmFjZT0iVGFob21hIiBzaXplPSI1IiBjb2xvcj0iIzgwMDA4MCI+DQo8YSBocmVmPSJt YWlsdG86ZGFyaW9iZXJAc3BlZWR5LmNvbS5hciI+ZGFyaW9iZXJAc3BlZWR5LmNvbS5hcjwv YT48L2ZvbnQ+PC9zdHJvbmc+PC9wPg0KDQo8L2JvZHk+DQoNCjwvaHRtbD4= ------=_NextPart_000_001__2771585_76741,02-- From mark.favas@csiro.au Sat Oct 6 03:43:22 2001 From: mark.favas@csiro.au (Mark Favas) Date: Sat, 06 Oct 2001 10:43:22 +0800 Subject: [Python-Dev] Painful death in debug build Message-ID: <3BBE6FCA.DEDF4DD6@csiro.au> Following Martin, and configuring current CVS with --with-pydebug on Tru64 Unix (wonder what HP will call it... <wink>), I can't get test_descr.py to crash either. Tried also with --with-pymalloc and different compiler optimizations. Cheers, Mark -- Mark Favas - m.favas@per.dem.csiro.au CSIRO, Private Bag No 5, Wembley, Western Australia 6913, AUSTRALIA From skip@pobox.com (Skip Montanaro) Sat Oct 6 03:58:03 2001 From: skip@pobox.com (Skip Montanaro) (Skip Montanaro) Date: Fri, 5 Oct 2001 21:58:03 -0500 Subject: [Python-Dev] Painful death in debug build In-Reply-To: <LNBBLJKPBEHFEDALKOLCKEJGLPAA.tim.one@home.com> References: <LNBBLJKPBEHFEDALKOLCKEJGLPAA.tim.one@home.com> Message-ID: <15294.29499.710313.179455@beluga.mojam.com> Tim> test_descr.py crashes under the debug build now. I've got too many Tim> things piled on my stack to look into it right now, so it would be Tim> good if someone could try to feel debug pain on Linux too. Works for me. (Mandrake 8.0, GCC 3.0.1) test_fcntl failed during the first pass of "make test", but succeeded with the -O test. When I ran it manually, it also succeeded. When I ran make quicktest it crashed in the non -O run as well. Don't know if it's related or not. I haven't seen this before. test test_fcntl crashed -- exceptions.IOError: [Errno 22] Invalid argument Skip From tim.one@home.com Sat Oct 6 04:01:22 2001 From: tim.one@home.com (Tim Peters) Date: Fri, 5 Oct 2001 23:01:22 -0400 Subject: [Python-Dev] RE: Painful death in debug build In-Reply-To: <200110052354.f95Ns3007683@mira.informatik.hu-berlin.de> Message-ID: <LNBBLJKPBEHFEDALKOLCGEJPLPAA.tim.one@home.com> [Martin v. Loewis] > Configuring Python with --with-pydebug, I cannot get test_descr to > crash on Linux, using the current CVS. I believe that <wink>. It's not dying with a memory fault, it's dying because code in MS's debug libraries is proactively searching for memory corruption, finding it, then deliberately causing a crash as its way of spelling "hey! your program is insane". I expect you'd need something like Insure++ or efence to have a good chance of catching it on Linux. Sooner or later it will manifest as insane behavior in the release build -- this is "wild store" stuff, like bad pointer arithmetic, or out-of-bounds array indexing. Resulting problems usually occur long after the bad store, and can be symptom-free by luck (until you're doing a demo <0.9 wink>). From fdrake@acm.org Sat Oct 6 04:14:22 2001 From: fdrake@acm.org (Fred L. Drake, Jr.) Date: Fri, 5 Oct 2001 23:14:22 -0400 Subject: [Python-Dev] RE: Painful death in debug build In-Reply-To: <LNBBLJKPBEHFEDALKOLCGEJPLPAA.tim.one@home.com> References: <200110052354.f95Ns3007683@mira.informatik.hu-berlin.de> <LNBBLJKPBEHFEDALKOLCGEJPLPAA.tim.one@home.com> Message-ID: <15294.30478.165439.151351@grendel.zope.com> Tim Peters writes: > spelling "hey! your program is insane". I expect you'd need something like > Insure++ or efence to have a good chance of catching it on Linux. Sooner or I've started an Electric Fence run on a Mandrake 7.2 box, running the regression tests. -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> PythonLabs at Zope Corporation From fdrake@acm.org Sat Oct 6 04:38:22 2001 From: fdrake@acm.org (Fred L. Drake, Jr.) Date: Fri, 5 Oct 2001 23:38:22 -0400 Subject: [Python-Dev] RE: Painful death in debug build In-Reply-To: <LNBBLJKPBEHFEDALKOLCGEJPLPAA.tim.one@home.com> References: <200110052354.f95Ns3007683@mira.informatik.hu-berlin.de> <LNBBLJKPBEHFEDALKOLCGEJPLPAA.tim.one@home.com> Message-ID: <15294.31918.124898.851098@grendel.zope.com> Given that Guido's already done the efence run, I'll ignore the weird result I'm getting. ;-) On a laptop running Mandrake 7.2, Python configured --with-pydebug, and lots of free memory, I get this: beowolf(.../python/debug); LD_PRELOAD=/usr/lib/libefence.so.0.0 ./python ../Lib/test/regrtest.py -l Electric Fence 2.2.0 Copyright (C) 1987-1999 Bruce Perens <bruce@perens.com> Adding parser accelerators ... Done. test_grammar test_opcodes test_operations test_builtin test_exceptions test_types test_MimeWriter test_StringIO test___all__ ElectricFence Exiting: mmap() failed: Cannot allocate memory [255] beowolf(.../python/debug); free total used free shared buffers cached Mem: 523808 204256 319552 0 29112 116824 -/+ buffers/cache: 58320 465488 Swap: 1050800 0 1050800 -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> PythonLabs at Zope Corporation From fdrake@acm.org Sat Oct 6 04:44:48 2001 From: fdrake@acm.org (Fred L. Drake, Jr.) Date: Fri, 5 Oct 2001 23:44:48 -0400 Subject: [Python-Dev] Need a way to test for 8-bit-or-unicode-string In-Reply-To: <200110052101.RAA02367@cj20424-a.reston1.va.home.com> References: <200110051436.KAA24983@cj20424-a.reston1.va.home.com> <3BBDD79D.C26D1646@lemburg.com> <200110051557.LAA25625@cj20424-a.reston1.va.home.com> <15293.57386.311536.729065@anthem.wooz.org> <3BBDE670.52DFB7F0@lemburg.com> <15293.59431.1516.201622@anthem.wooz.org> <3BBDFA29.A5A4DCBA@lemburg.com> <200110052101.RAA02367@cj20424-a.reston1.va.home.com> Message-ID: <15294.32304.20074.289952@grendel.zope.com> Guido van Rossum writes: > Followup question: should I also implement issubclass(X, (A, B, C...))? > > It's consistent to do so, but it's more work (issubclass is a bit > murkier due to the "abstract subclass" support), and I'm not sure that > it has the same benefits... They're similar, but they're usage is not. Let's keep issubclass() simple until we know there's a reason to change it. -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> PythonLabs at Zope Corporation From nas@python.ca Sat Oct 6 05:44:55 2001 From: nas@python.ca (Neil Schemenauer) Date: Fri, 5 Oct 2001 21:44:55 -0700 Subject: [Python-Dev] RE: Painful death in debug build In-Reply-To: <15294.31918.124898.851098@grendel.zope.com>; from fdrake@acm.org on Fri, Oct 05, 2001 at 11:38:22PM -0400 References: <200110052354.f95Ns3007683@mira.informatik.hu-berlin.de> <LNBBLJKPBEHFEDALKOLCGEJPLPAA.tim.one@home.com> <15294.31918.124898.851098@grendel.zope.com> Message-ID: <20011005214455.A26548@glacier.arctrix.com> Fred L. Drake, Jr. wrote: > > Given that Guido's already done the efence run, I'll ignore the > weird result I'm getting. ;-) > On a laptop running Mandrake 7.2, Python configured --with-pydebug, > and lots of free memory, I get this: > > > beowolf(.../python/debug); LD_PRELOAD=/usr/lib/libefence.so.0.0 ./python ../Lib/test/regrtest.py -l ... > ElectricFence Exiting: mmap() failed: Cannot allocate memory Using electric fence requires huge amounts of (virtual) memory. Every allocation requires two pages of memory. I doubt running the entire test suite would ever work. I'm having trouble running just test_descr alone with all the tests commented out! I think I'm running into MAX_MAP_COUNT (which defaults to 64k). I've just grabbed Linux 2.4.10-ac7 which add an AVL tree for VMAs and a dynamically configurable max_map_count. We'll see how that goes. Someone should really write a portable debug version of malloc and free for Python that works like the Microsoft implementation. It can't be too hard to lay down poison before and after the hunk of memory on allocation and then check it on deallocation. Neil From tim.one@home.com Sat Oct 6 06:02:09 2001 From: tim.one@home.com (Tim Peters) Date: Sat, 6 Oct 2001 01:02:09 -0400 Subject: [Python-Dev] RE: Painful death in debug build In-Reply-To: <20011005214455.A26548@glacier.arctrix.com> Message-ID: <LNBBLJKPBEHFEDALKOLCEEKDLPAA.tim.one@home.com> [Neil Schemenauer] > Using electric fence requires huge amounts of (virtual) memory. Every > allocation requires two pages of memory. Yikes! [NealS] > I doubt running the entire test suite would ever work. I'm having > trouble running just test_descr alone with all the tests commented out! FWIW, I can provoke the bug by running test_descr.inherits() alone: C:\Code\python\PCbuild>python_d Adding parser accelerators ... Done. Python 2.2a4+ (#24, Oct 5 2001, 18:16:47) [MSC 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> from test import test_descr [21862 refs] >>> test_descr.inherits() Testing inheritance from basic types... And then it blows up. > I think I'm running into MAX_MAP_COUNT (which defaults to 64k). > > I've just grabbed Linux 2.4.10-ac7 which add an AVL tree for VMAs and a > dynamically configurable max_map_count. We'll see how that goes. You may as well rewrite the kernel while you're at it <wink>. > Someone should really write a portable debug version of malloc and free > for Python that works like the Microsoft implementation. It can't be > too hard to lay down poison before and after the hunk of memory on > allocation and then check it on deallocation. It's extremely useful for establishing the presence of off-by-one stores, but this is the first time I've seen this particular check trigger in Python. Another trick it plays is to fill free()'ed memory with a different unlikely byte pattern, then delay recycling the memory. When things blow up this "dead landfill" value is amazingly often present (indicating that someone is still trying to read from free'd memory). The third trick is that newly malloc'ed memory is filled with a third "clean landfill" byte, which helps catch use of uninitialized memory. All the special byte values are picked so that they're unlikely to yield valid addresses, or integers. In bang for the buck, these tricks are huge wins. if-only-they-pointed-to-the-true-causes-too-ly y'rs - tim From tim.one@home.com Sat Oct 6 06:10:02 2001 From: tim.one@home.com (Tim Peters) Date: Sat, 6 Oct 2001 01:10:02 -0400 Subject: FW: [PythonLabs] Re: [Python-Dev] RE: Painful death in debug build Message-ID: <LNBBLJKPBEHFEDALKOLCAEKELPAA.tim.one@home.com> FYI, for anyone else looking at this. Guido tricked me into sending this to an internal PythonLabs address. -----Original Message----- From: Zope.Com Python Labs Sent: Saturday, October 06, 2001 12:47 AM To: Guido van Rossum Cc: Zope.Com Python Labs Subject: [PythonLabs] Re: [Python-Dev] RE: Painful death in debug build [Guido] > I tried efence on Linux (which traps bad stores using VM tricks) and > couldn't get it to fail either. Thanks for trying! I'll pursue here then. VM tricks may not be enough, unless they're very fine-grained. The MS debug malloc overallocates by 8 bytes, 4 on each side of the blocks it returns. These get filled with 0xFD bytes. When the debug free() gets a block back, it first checks these bytes. The four before the block are OK (still 0xFD). The first two bytes after the block have been zeroed, but the two bytes after those are still 0xFD. So who's storing 0 into a short via a computed address suffering an off-by-one error <0.9 wink>? (*) When I crank up enough energy, it should be more-or-less straightforward to set a data breakpoint on one of the byte addresses getting corrupted, and so catch the culprit in the act (the potential problem is that memory gets recycled, and on occasion I've had to give up on this approach after wading thru 100s of false alarms). (*) Hmm! Cogitating: the traceback pointed at a subclass of long, and the internal representation of longs is indeed one of the few places Python uses an array of shorts (Unicode is another -- others?). A dollar bets this will prove to be relevant. From tim.one@home.com Sat Oct 6 06:32:44 2001 From: tim.one@home.com (Tim Peters) Date: Sat, 6 Oct 2001 01:32:44 -0400 Subject: : [Python-Dev] RE: Painful death in debug build In-Reply-To: <LNBBLJKPBEHFEDALKOLCAEKELPAA.tim.one@home.com> Message-ID: <LNBBLJKPBEHFEDALKOLCCEKFLPAA.tim.one@home.com> This is enough to reproduce the problem under a debug Windows build: --------------------------------------------------------------- class X(long): pass x = X(0xffffL) print "before del" del x print "after del" --------------------------------------------------------------- It does not fail if the instance is created via x = X(0x7fffL) instead. It does fail on x = X(0x8000L) The relevant difference is that Python uses 15-bit "digits" internally for longs, so 0x8000 may be the smallest value that will cause it to allocate more memory than already comes for free with the _longobject header: struct _longobject { PyObject_HEAD int ob_size; digit ob_digit[1]; }; but-too-tired-to-do-more-now-ly y'rs - tim From fdrake@acm.org Sat Oct 6 07:09:45 2001 From: fdrake@acm.org (Fred L. Drake) Date: Sat, 6 Oct 2001 02:09:45 -0400 (EDT) Subject: [Python-Dev] [development doc updates] Message-ID: <20011006060945.4C32828697@beowolf.digicool.com> The development version of the documentation has been updated: http://python.sourceforge.net/devel-docs/ Updates to add ReferenceError to the standard exceptions. From tim.one@home.com Sat Oct 6 09:35:01 2001 From: tim.one@home.com (Tim Peters) Date: Sat, 6 Oct 2001 04:35:01 -0400 Subject: : [Python-Dev] RE: Painful death in debug build In-Reply-To: <LNBBLJKPBEHFEDALKOLCCEKFLPAA.tim.one@home.com> Message-ID: <LNBBLJKPBEHFEDALKOLCMEKILPAA.tim.one@home.com> [Tim] > This is enough to reproduce the problem under a debug Windows build: > > --------------------------------------------------------------- > class X(long): > pass > > x = X(0xffffL) > print "before del" > del x > print "after del" > --------------------------------------------------------------- > > It does not fail if the instance is created via > > x = X(0x7fffL) > > instead. It does fail on > > x = X(0x8000L) > > The relevant difference is that Python uses 15-bit "digits" internally > for longs, so 0x8000 may be the smallest value that will cause it to > allocate more memory than already comes for free with the _longobject > header Now I'm in: PyObject * PyType_GenericAlloc(PyTypeObject *type, int nitems) { #define PTRSIZE (sizeof(PyObject *)) int size; PyObject *obj; /* Inline PyObject_New() so we can zero the memory */ That comment doesn't seem to make much sense; the code following does a whole bunch that PyObject_New() doesn't do. size = _PyObject_VAR_SIZE(type, nitems); Then size = basicsize + 2 * itemsize = 26 + 2*2 = 30. /* Round up size, if necessary, so we fully zero out __dict__ */ if (type->tp_itemsize % PTRSIZE != 0) { size += PTRSIZE - 1; size /= PTRSIZE; size *= PTRSIZE; } I don't understand what that comment is trying to say; the code following it bumps size up to 32, and that turns out to be "the problem": if (PyType_IS_GC(type)) { obj = _PyObject_GC_Malloc(type, nitems); } else { obj = PyObject_MALLOC(size); } The subtype is a GC type, so _PyObject_GC_Malloc allocates sizeof(PyGC_Head) + (size_t)_PyObject_VAR_SIZE(tp, size) = 12 + 30 = 42 bytes. Note that the second term is the 30 originally computed for size, not the 32 that size was later bumped up to. if (obj == NULL) return PyErr_NoMemory(); Nope. memset(obj, '\0', size); Bingo! obj is at offset 12 from the allocated block, and size is 32, so we zero out the last 32 of the trailing 30 (=42-12) bytes allocated; i.e., we zero out two bytes beyond the end of the allocated block. Everything else follows from that. Seems pretty clear that this doesn't have anything specific to do with longs. Yup: >>> class X(str): ... pass ... [6960 refs] >>> x = X("ab") [6969 refs] >>> x = 1 That also blows up. Since I'm not sure what the roundup of size is trying to accomplish, I'm not sure what to do about it. If I ignore the comment, I *imagine* we want subtype pointer fields (like tp_dict) following the base-type prefix to be naturally aligned for the architecture, and then padding makes sense to me. From guido@python.org Sat Oct 6 18:49:13 2001 From: guido@python.org (Guido van Rossum) Date: Sat, 06 Oct 2001 13:49:13 -0400 Subject: [Python-Dev] RE: Painful death in debug build In-Reply-To: Your message of "Fri, 05 Oct 2001 21:44:55 PDT." <20011005214455.A26548@glacier.arctrix.com> References: <200110052354.f95Ns3007683@mira.informatik.hu-berlin.de> <LNBBLJKPBEHFEDALKOLCGEJPLPAA.tim.one@home.com> <15294.31918.124898.851098@grendel.zope.com> <20011005214455.A26548@glacier.arctrix.com> Message-ID: <200110061749.NAA09260@cj20424-a.reston1.va.home.com> > Using electric fence requires huge amounts of (virtual) memory. Every > allocation requires two pages of memory. I doubt running the entire > test suite would ever work. I'm having trouble running just test_descr > alone with all the tests commented out! I think I'm running into > MAX_MAP_COUNT (which defaults to 64k). The trick is to compile it to a .pyc file without efence. The compiler is vastly malloc-happy. Once you have the .pyc, test_descr runs to completion without errors using efence. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Sat Oct 6 19:42:55 2001 From: guido@python.org (Guido van Rossum) Date: Sat, 06 Oct 2001 14:42:55 -0400 Subject: : [Python-Dev] RE: Painful death in debug build In-Reply-To: Your message of "Sat, 06 Oct 2001 04:35:01 EDT." <LNBBLJKPBEHFEDALKOLCMEKILPAA.tim.one@home.com> References: <LNBBLJKPBEHFEDALKOLCMEKILPAA.tim.one@home.com> Message-ID: <200110061842.OAA09418@cj20424-a.reston1.va.home.com> So the problem is that one size calculation rounds up to multiples of a pointer size and the other doesn't; unfortunately the one that doesn't gets passed to malloc. The reason for the rounding up --a recent addition-- is that when you derive a subclass from tuple, string or long, the __dict__ of the subclass instance (and its weakref list head) are allocated after the variable-length portion of the instance. I propose to fix the other size calculation to also round up. I'll leave it to you to fix this, since you've almost nailed this already (and since I'm not feeling so great right now). --Guido van Rossum (home page: http://www.python.org/~guido/) From nas@python.ca Sat Oct 6 21:50:26 2001 From: nas@python.ca (Neil Schemenauer) Date: Sat, 6 Oct 2001 13:50:26 -0700 Subject: [Python-Dev] RE: Painful death in debug build In-Reply-To: <200110061749.NAA09260@cj20424-a.reston1.va.home.com>; from guido@python.org on Sat, Oct 06, 2001 at 01:49:13PM -0400 References: <200110052354.f95Ns3007683@mira.informatik.hu-berlin.de> <LNBBLJKPBEHFEDALKOLCGEJPLPAA.tim.one@home.com> <15294.31918.124898.851098@grendel.zope.com> <20011005214455.A26548@glacier.arctrix.com> <200110061749.NAA09260@cj20424-a.reston1.va.home.com> Message-ID: <20011006135026.A27373@glacier.arctrix.com> Guido van Rossum wrote: > The trick is to compile it to a .pyc file without efence. The > compiler is vastly malloc-happy. Once you have the .pyc, test_descr > runs to completion without errors using efence. I figured that out after watching the process grow to 800 MB. :-) Neil From aahz@rahul.net Sun Oct 7 17:00:31 2001 From: aahz@rahul.net (Aahz Maruch) Date: Sun, 7 Oct 2001 09:00:31 -0700 (PDT) Subject: [Python-Dev] Need a way to test for 8-bit-or-unicode-string In-Reply-To: <15293.53573.251652.664773@anthem.wooz.org> from "Barry A. Warsaw" at Oct 05, 2001 11:27:01 AM Message-ID: <20011007160032.39A87E8C6@waltz.rahul.net> [I know we've already reached agreement, but I feel the need to stick my oar in.] Barry A. Warsaw wrote: > > NS> Perhaps we could extend isinstance(). How about > > NS> isinstance(x, str, unicode) > > NS> or > > NS> isinstance(x, (str, unicode)) > > NS> This is a common problem not limited to string types. I often > NS> want to test if something is a tuple or a list for example. > > That's an interesting idea, but please use the former signature, not > the latter. Ewww, gross, spit spit spit. ;-) At least with the latter, you can do: stringtype = (str, unicode) isinstance(x, stringtype) Hmmmm... Or can you? -- --- Aahz (@pobox.com) Hugs and backrubs -- I break Rule 6 <*> http://www.rahul.net/aahz/ Androgynous poly kinky vanilla queer het Pythonista We must not let the evil of a few trample the freedoms of the many. From gstein@lyra.org Sun Oct 7 23:17:47 2001 From: gstein@lyra.org (Greg Stein) Date: Sun, 7 Oct 2001 15:17:47 -0700 Subject: [Python-Dev] imputil and modulefinder replacements In-Reply-To: <200110051306.JAA24439@cj20424-a.reston1.va.home.com>; from guido@python.org on Fri, Oct 05, 2001 at 09:06:32AM -0400 References: <3BB204E3.23339.3A9F068C@localhost> <064a01c14d84$3ed0fcf0$e000a8c0@thomasnotebook> <200110051306.JAA24439@cj20424-a.reston1.va.home.com> Message-ID: <20011007151746.A4216@lyra.org> On Fri, Oct 05, 2001 at 09:06:32AM -0400, Guido van Rossum wrote: > > It seems noone cares about this. imputil is in the distribution, > > but is it really 'official'? > > I want to care, but I'm already spread too thin. I don't find > 'imputil' particularly official -- but Greg Stein pushed hard to get > it in the distribution. Don't blame me :-) I thought it was much better than ihooks, and a good number of people wanted to see it in the distro. I wrote imputil to solve some of my problems, and it was a great step forward over the ihooks stuff. It was reasonably clean, very easy to extend, and helped to clarify the existing import and package mechanisms (by virtue of needing to replace them). Gordon picked up imputil and did a lot more work with it than I ever did. At this point, if he says that iu4 is better than imputil, then I'd be a mind to trust that :-) I'm just happy that my imputil was able to show a way for even better improvements. Personally, I don't have a particular attachment to preferring it over Gordon's new iu4. iu4 probably needs a bit wider usage to see whether *other* people can use it like Gordon :-), but it looks like he has documented it quite well. Cheers, -g -- Greg Stein, http://www.lyra.org/ From gstein@lyra.org Mon Oct 8 00:59:39 2001 From: gstein@lyra.org (Greg Stein) Date: Sun, 7 Oct 2001 16:59:39 -0700 Subject: [Python-Dev] Integrating Expat In-Reply-To: <200109301453.QAA21436@paros.informatik.hu-berlin.de>; from loewis@informatik.hu-berlin.de on Sun, Sep 30, 2001 at 04:53:06PM +0200 References: <200109301453.QAA21436@paros.informatik.hu-berlin.de> Message-ID: <20011007165939.B4216@lyra.org> On Sun, Sep 30, 2001 at 04:53:06PM +0200, Martin von Loewis wrote: > [I know I've asked this before, but Fred wanted me to ask it again :-] > > What do you think about an integration of Expat into Python, to be > always able to build pyexpat (and with the same version also)? > Which version of Expat would you use? Would you put the expat files > into a separate directory, or all into modules? > > Here is my proposal: Integrate Expat 2.95.2 for release together with > Python 2.2; into an expat subdirectory of Modules (taking only the lib > files of expat). > > This would affect build procedures on all targets; in particular, > pyexpat would not link to a shared expat DLL, but incorporate the > object files. Speaking from the experience of bundling Expat directly into the Apache binaries (also using a subset of the original source) ... I think bundling the sources is fine, but it should *ONLY* be a fallback if you do not find the Expat library installed on the system. *ALWAYS* link against a system-installed library first. We ran into a problem that has bothered some Perl users for a long while now. Specifically: Apache 1.3 would get loaded and export the Expat symbols to the rest of the process. Any third-party module that was built *against Apache* (obviously the case since they are Apache modules) and needed Expat would immediately resolve upon loading and be happy. But! What we ran into is mod_perl (linked against Apache) running a Perl script which, in turn, loaded XML::Parsers::Expat. That Perl module linked against *Expat*, not Apache (it is a standard module and has nothing to do with Apache). Well... when the Perl module was loaded, you now had *two* sets of Expat symbols in the process space. Segfaults, bugs, and madness ensued. I just made some fixes this past week to Apache 1.3 to fix the situation somewhat. The basic answer is to always grab a system (.so) library when possible. When the shared lib is present, then both Apache and XML::Parsers::Expat would link against the same thing about loading. And Apache still has the feature of exposing XML to its third-party modules. This situation could easily happen to Python, too. Imagine building Expat directly into pyexpat. Some Python script loads pyexpat and the Expat symbols come with it. Now, some *other* module is loaded and dynamically links against /usr/lib/libexpat.so. Now you have *two* sets of Expat symbols and crashes are going to start happening. -1 on *always* using bundles sources -- they should only be a fallback. +1 on including it as a fallback. Cheers, -g -- Greg Stein, http://www.lyra.org/ From tim.one@home.com Mon Oct 8 08:36:19 2001 From: tim.one@home.com (Tim Peters) Date: Mon, 8 Oct 2001 03:36:19 -0400 Subject: [Python-Dev] Need a way to test for 8-bit-or-unicode-string In-Reply-To: <20011007160032.39A87E8C6@waltz.rahul.net> Message-ID: <LNBBLJKPBEHFEDALKOLCIENDLPAA.tim.one@home.com> [Aahz] > At least with the latter, you can do: > > stringtype = (str, unicode) > isinstance(x, stringtype) > > Hmmmm... Or can you? Yes -- it's the value of the expression that counts, not how it's spelled. You can even nest these tuples: >>> int_types = int, long >>> file_types = file, >>> def fori(x): ... return isinstance(x, (int_types, file_types)) ... >>> fori(43) 1 >>> fori(43L) 1 >>> import sys >>> fori(sys.stdin) 1 >>> fori(43.0) 0 >>> From tim.one@home.com Mon Oct 8 08:54:59 2001 From: tim.one@home.com (Tim Peters) Date: Mon, 8 Oct 2001 03:54:59 -0400 Subject: [Python-Dev] Need a way to test for 8-bit-or-unicode-string In-Reply-To: <200110051436.KAA24983@cj20424-a.reston1.va.home.com> Message-ID: <LNBBLJKPBEHFEDALKOLCMENELPAA.tim.one@home.com> [Guido] > ... > [*] For a while I toyed with the idea of calling the abstract base > class 'string', and hacking import so that sys.modules['string'] is > the string class. The abstract base class should then have methods > that invoke the concrete implementations, so that string.split(s) > would be the same as s.split(). > ... > Unfortunately this broke down when I realized that the signature of > string.join() is wrong for the string module: the string.join function > is string.join(sequence, stringobject) while the signature of the > string method is join(stringobject, sequence). > ... > (Hm, it could be saved by making string.join() accept the arguments > in either order. Gross. :-) Luckily, it's also hopeless: >>> string.join('abc', ' - ') 'a - b - c' >>> That is, since strings are also sequences, there's no way to disambiguate the pass-2-strings case. Although, if you ask me, it's obvious I wanted the result shown instead of ' abc-abc ' <wink>. From martin@loewis.home.cs.tu-berlin.de Mon Oct 8 09:03:54 2001 From: martin@loewis.home.cs.tu-berlin.de (Martin v. Loewis) Date: Mon, 8 Oct 2001 10:03:54 +0200 Subject: [Python-Dev] Integrating Expat Message-ID: <200110080803.f9883sP01692@mira.informatik.hu-berlin.de> > I think bundling the sources is fine, but it should *ONLY* be a fallback > if you do not find the Expat library installed on the system. *ALWAYS* > link against a system-installed library first. I'm not so sure about that. There are many problems doing that: - if there is no shared libexpat, a static one is only usable if it is compiled either as PIC, or if pyexpat is statically linked into the Python interpreter. These conditions are impossible to test. - if the installed version of expat is older than the one we ship, I think Python should still use the included one. I think the problem you had with Apache does not apply to Python: > Apache 1.3 would get loaded and export the Expat symbols to the rest > of the process. In Python, symbols of one extension module are not exported to anything else. We would not build a libexpat.so (not sure whether Apache did), and nobody would link with pyexpat.so. Furthermore, dynloading pyexpat.so will use RTLD_LOCAL, unless that is overridden by the application (and you really have to know what you're doing when changing the dlopen flags). > Now, some *other* module is loaded and dynamically links against > /usr/lib/libexpat.so. Now you have *two* sets of Expat symbols and > crashes are going to start happening. No, that can't happen. pyexpat.so does not export its symbols (due to RTLD_LOCAL), so the other module will still use libexpat.so. Regards, Martin From mal@lemburg.com Mon Oct 8 09:40:04 2001 From: mal@lemburg.com (M.-A. Lemburg) Date: Mon, 08 Oct 2001 10:40:04 +0200 Subject: [Python-Dev] Need a way to test for 8-bit-or-unicode-string References: <200110051436.KAA24983@cj20424-a.reston1.va.home.com> <3BBDD79D.C26D1646@lemburg.com> <200110051557.LAA25625@cj20424-a.reston1.va.home.com> <15293.57386.311536.729065@anthem.wooz.org> <3BBDE670.52DFB7F0@lemburg.com> <15293.59431.1516.201622@anthem.wooz.org> <3BBDFA29.A5A4DCBA@lemburg.com> <200110052101.RAA02367@cj20424-a.reston1.va.home.com> Message-ID: <3BC16664.5BE1E0B8@lemburg.com> Guido van Rossum wrote: > > Now that the introduction of a class hierarchy is dead, Is it ? I didn't see any messages on the list saying that the idea is dead -- only that a PEP is needed and that it cannot be done for 2.2. > I'd like to > move forward with the isinstance(x, (A, B, C...)) patch. > > Followup question: should I also implement issubclass(X, (A, B, C...))? > > It's consistent to do so, but it's more work (issubclass is a bit > murkier due to the "abstract subclass" support), and I'm not sure that > it has the same benefits... -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Consulting & Company: http://www.egenix.com/ Python Software: http://www.lemburg.com/python/ From mwh@python.net Mon Oct 8 10:55:33 2001 From: mwh@python.net (Michael Hudson) Date: 08 Oct 2001 05:55:33 -0400 Subject: [Python-Dev] Need a way to test for 8-bit-or-unicode-string In-Reply-To: aahz@rahul.net's message of "Sun, 7 Oct 2001 09:00:31 -0700 (PDT)" References: <20011007160032.39A87E8C6@waltz.rahul.net> Message-ID: <2mk7y6lalm.fsf@starship.python.net> aahz@rahul.net (Aahz Maruch) writes: > [I know we've already reached agreement, but I feel the need to stick my > oar in.] > > Barry A. Warsaw wrote: > > > > NS> Perhaps we could extend isinstance(). How about > > > > NS> isinstance(x, str, unicode) > > > > NS> or > > > > NS> isinstance(x, (str, unicode)) > > > > NS> This is a common problem not limited to string types. I often > > NS> want to test if something is a tuple or a list for example. > > > > That's an interesting idea, but please use the former signature, not > > the latter. > > Ewww, gross, spit spit spit. ;-) > > At least with the latter, you can do: > > stringtype = (str, unicode) > isinstance(x, stringtype) > > Hmmmm... Or can you? Yes, and with the former, you can do stringtype = (str, unicode) isinstance(x, *stringtype) not-that-I-care-ly y'rs M. -- Indeed, when I design my killer language, the identifiers "foo" and "bar" will be reserved words, never used, and not even mentioned in the reference manual. Any program using one will simply dump core without comment. Multitudes will rejoice. -- Tim Peters, 29 Apr 1998 From gstein@lyra.org Mon Oct 8 12:29:32 2001 From: gstein@lyra.org (Greg Stein) Date: Mon, 8 Oct 2001 04:29:32 -0700 Subject: [Python-Dev] Integrating Expat In-Reply-To: <200110080803.f9883sP01692@mira.informatik.hu-berlin.de>; from martin@loewis.home.cs.tu-berlin.de on Mon, Oct 08, 2001 at 10:03:54AM +0200 References: <200110080803.f9883sP01692@mira.informatik.hu-berlin.de> Message-ID: <20011008042932.H4216@lyra.org> On Mon, Oct 08, 2001 at 10:03:54AM +0200, Martin v. Loewis wrote: > > I think bundling the sources is fine, but it should *ONLY* be a fallback > > if you do not find the Expat library installed on the system. *ALWAYS* > > link against a system-installed library first. > > I'm not so sure about that. There are many problems doing that: > > - if there is no shared libexpat, a static one is only usable if it is > compiled either as PIC, or if pyexpat is statically linked into > the Python interpreter. These conditions are impossible to test. What? I've never seen a .a installed on a system that isn't usable. What makes Python modules so special that they could not use a library? I find your statement rather surprising, and I'm quite doubtful of the issue. I've linked .a files into .so modules often. Never a problem. > - if the installed version of expat is older than the one we ship, > I think Python should still use the included one. Every time you use the included one, you run into the possibility of symbol conflict problems. If you refuse to use the one installed in the system, then you increase the chance of problems. Granted: there isn't much you can do if pyexpat *requires* something newer than what is on the system (does it require the latest? or can pyexpat work with all versions of expat?). > I think the problem you had with Apache does not apply to Python: > > > Apache 1.3 would get loaded and export the Expat symbols to the rest > > of the process. > > In Python, symbols of one extension module are not exported to > anything else. Eh? Again: what makes Python so special? How can you guarantee this rule? On *all* platforms? > We would not build a libexpat.so (not sure whether > Apache did), We never built a .so. The library was linked right into the Apache httpd executable. > and nobody would link with pyexpat.so. Furthermore, > dynloading pyexpat.so will use RTLD_LOCAL, unless that is overridden > by the application (and you really have to know what you're doing when > changing the dlopen flags). This argument doesn't hold. dlopen() is not used on every platform. Thus, you cannot specify RTLD_LOCAL on all platforms, thus you cannot guarantee that symbols in a library will remain private. > > Now, some *other* module is loaded and dynamically links against > > /usr/lib/libexpat.so. Now you have *two* sets of Expat symbols and > > crashes are going to start happening. > > No, that can't happen. pyexpat.so does not export its symbols (due to > RTLD_LOCAL), so the other module will still use libexpat.so. See above. Incorporating Expat directly into Python will definitely be "nice", but I've been through this already. It isn't pretty. If that shared library exists, then use the darned thing, or you're gonna get burned. Cheers, -g -- Greg Stein, http://www.lyra.org/ From guido@python.org Mon Oct 8 14:14:17 2001 From: guido@python.org (Guido van Rossum) Date: Mon, 08 Oct 2001 09:14:17 -0400 Subject: [Python-Dev] Need a way to test for 8-bit-or-unicode-string In-Reply-To: Your message of "Mon, 08 Oct 2001 10:40:04 +0200." <3BC16664.5BE1E0B8@lemburg.com> References: <200110051436.KAA24983@cj20424-a.reston1.va.home.com> <3BBDD79D.C26D1646@lemburg.com> <200110051557.LAA25625@cj20424-a.reston1.va.home.com> <15293.57386.311536.729065@anthem.wooz.org> <3BBDE670.52DFB7F0@lemburg.com> <15293.59431.1516.201622@anthem.wooz.org> <3BBDFA29.A5A4DCBA@lemburg.com> <200110052101.RAA02367@cj20424-a.reston1.va.home.com> <3BC16664.5BE1E0B8@lemburg.com> Message-ID: <200110081314.JAA30586@cj20424-a.reston1.va.home.com> > Guido van Rossum wrote: > > > > Now that the introduction of a class hierarchy is dead, > > Is it ? I didn't see any messages on the list saying that the > idea is dead -- only that a PEP is needed and that it cannot > be done for 2.2. Sorry, I meant that the introduction of a hierarchy in 2.2 is dead. --Guido van Rossum (home page: http://www.python.org/~guido/) From jim@interet.com Mon Oct 8 16:09:40 2001 From: jim@interet.com (James C. Ahlstrom) Date: Mon, 08 Oct 2001 11:09:40 -0400 Subject: [Python-Dev] imputil and modulefinder replacements References: <3BBDCFC0.18517.68AF780F@localhost> Message-ID: <3BC1C1B4.5EAEECAE@interet.com> Gordon McMillan wrote: > With Thomas's changes to PyImport_ImportModuleEx, this is > largely taken care of (it gets routed through __import__). So > with the above bootstrap code, C code will import from my > .pyz as well. I am not sure if you are saying that code already exists to import from zip archives, or that iu4.py should be used as the basis. I think the best approach is for me to write up a PEP based on the discussions so far, and let everyone criticize. The proposal is only to import from zip archives, not whether to replace imputil.py with iu4.py. I will submit a PEP after I write some code. I often find that my initial design is poor once I implement it. So expect the PEP in a week or so. This won't make it into 2.2, so there is no rush, right? Jim From martin@loewis.home.cs.tu-berlin.de Tue Oct 9 02:55:15 2001 From: martin@loewis.home.cs.tu-berlin.de (Martin v. Loewis) Date: Tue, 9 Oct 2001 03:55:15 +0200 Subject: [Python-Dev] Integrating Expat In-Reply-To: <20011008042932.H4216@lyra.org> (message from Greg Stein on Mon, 8 Oct 2001 04:29:32 -0700) References: <200110080803.f9883sP01692@mira.informatik.hu-berlin.de> <20011008042932.H4216@lyra.org> Message-ID: <200110090155.f991tFW04878@mira.informatik.hu-berlin.de> > What? I've never seen a .a installed on a system that isn't usable. What > makes Python modules so special that they could not use a library? I find > your statement rather surprising, and I'm quite doubtful of the issue. > > I've linked .a files into .so modules often. Never a problem. It happens on Solaris with gcc as the compiler and /usr/ccs/bin/ld as the linker. Watch this pandora loewis 39 ( ~/tmp ) > cat a.c int i; int f(){return i;} pandora loewis 40 ( ~/tmp ) > gcc -c a.c pandora loewis 41 ( ~/tmp ) > gcc -o a.so -shared a.o Text relocation remains referenced against symbol offset in file i 0x4 a.o i 0x8 a.o ld: fatal: relocations remain against allocatable but non-writable sections collect2: ld returned 1 exit status The problem here is that gcc passes -z text to the linker, unless -mimpure-text is specified. ld(1) then complains if the code was not compiled with -fPIC/-KPIC, i.e. that relocations remain. Python is not special at all in this respect; that happens every time when you integrate a non-PIC static library (or object file) into a shared library, and - the system is Solaris (or probably any SysV-derived system) - the linker is the system linker - the compiler passes -z text to the linker (i.e. it is gcc) If the linker is binutils ld, it won't happen since the linker ignores -z text. If the compiler is SUNWspro, it won't happen because cc does not pass -z text by default. On a Linux system, it won't happen because relocations against the text segment are not considered a problem in the first place - COW will take care of that. Of course, relocations against the text segment *are* a problem: they cause the shared library not to be shared at run-time. I don't know whether the problem is specific to Python; perhaps we deal with more varying configurations than anybody else. Please see the bug reports #232783, #404322, and #405329 for prior reports of this issue. > > - if the installed version of expat is older than the one we ship, > > I think Python should still use the included one. > > Every time you use the included one, you run into the possibility of > symbol conflict problems. If you refuse to use the one installed in the > system, then you increase the chance of problems. How would such a conflict occur using the current mechanisms for dynamic loading in Python? > > I think the problem you had with Apache does not apply to Python: > > > > > Apache 1.3 would get loaded and export the Expat symbols to the rest > > > of the process. > > > > In Python, symbols of one extension module are not exported to > > anything else. > > Eh? Again: what makes Python so special? How can you guarantee this > rule? On *all* platforms? Unlike everybody else, Python does *not* use RTLD_GLOBAL in the flags passed to dlopen. This alone is a guarantee that shared libraries won't interfere with each other: Optionally, RTLD_GLOBAL may be or'ed with flag, in which case the external symbols defined in the library will be made available to subsequently loaded libraries. [from the Linux dlopen man page] It will work on all systems using dynload_shlib, plus it will work on Windows (where you have to explicitly specify the symbols to export and the libraries you import symbols from). Please also note the Misc/HISTORY item - Revert a new feature in Unix dynamic loading: for one or two revisions, modules were loaded using the RTLD_GLOBAL flag. It turned out to be a bad idea. The specific problem was that somebody was using a library that exposed an initsocket function, which conflicted with the init function of the socket module. So we cannot do much for systems that cannot protect independent shared libraries from each other. > This argument doesn't hold. dlopen() is not used on every platform. Thus, > you cannot specify RTLD_LOCAL on all platforms, thus you cannot guarantee > that symbols in a library will remain private. Systems that do not support local symbols in shared libraries have bigger problems than competing libexpat implementations. BTW, which of the systems in today's use do not have dlopen, and aren't Windows? [How does MacOS X operate in this area?] > Incorporating Expat directly into Python will definitely be "nice", > but I've been through this already. It isn't pretty. If that shared > library exists, then use the darned thing, or you're gonna get > burned. OTOH, I'm not looking forward to bug reports which turn out problems with obsolete libexpat installations. You have to judge the risks. Any user who is concerned about using the installed libexpat can still modify Modules/Setup to use that instead of the one coming with Python. Regards, Martin From gmcm@hypernet.com Tue Oct 9 03:26:04 2001 From: gmcm@hypernet.com (Gordon McMillan) Date: Mon, 8 Oct 2001 22:26:04 -0400 Subject: [Python-Dev] imputil and modulefinder replacements In-Reply-To: <3BC1C1B4.5EAEECAE@interet.com> Message-ID: <3BC227FC.32750.79A82249@localhost> [Gordon] > > With Thomas's changes to PyImport_ImportModuleEx, this is > > largely taken care of (it gets routed through __import__). So > > with the above bootstrap code, C code will import from my .pyz > > as well. [Jim Ahsltrom] > I am not sure if you are saying that code already exists to > import from zip archives, or that iu4.py should be used as > the basis. > Neither. I thought you were referring to the Py 1.5-timeframe problem of C code using PyImport_ImportModule* and bypassing all hooks. > I think the best approach is for me to write up a PEP based > on the discussions so far, and let everyone criticize. The > proposal is only to import from zip archives, not > whether to replace imputil.py with iu4.py. Does "importing from zip archives" really need a PEP? It's already quasi-official in that Jython and MacPython implement it. The problem it exposes (the fact that the C code makes it difficult to do for packages) is certainly a good topic for discussion and is part of the reason Greg wrote imputil and I wrote iu4. - Gordon From pedroni@inf.ethz.ch Tue Oct 9 12:15:08 2001 From: pedroni@inf.ethz.ch (Samuele Pedroni) Date: Tue, 09 Oct 2001 13:15:08 +0200 Subject: [Python-Dev] imputil and modulefinder replacements References: <3BC227FC.32750.79A82249@localhost> Message-ID: <3BC2DC3C.AEEB3379@inf.ethz.ch> Hi. > > > Does "importing from zip archives" really need a PEP? It's > already quasi-official in that Jython and MacPython implement > it. Personally, I think so, a PEP would be indeed a good thing, mostly to fix the details. AFAIK MacPython and Jython are already different about the details, at least in Jython the feature is still experimental so there is space for changes, don't know in general. regards, Samule Pedroni. From guido@python.org Tue Oct 9 12:15:57 2001 From: guido@python.org (Guido van Rossum) Date: Tue, 09 Oct 2001 07:15:57 -0400 Subject: [Python-Dev] imputil and modulefinder replacements In-Reply-To: Your message of "Tue, 09 Oct 2001 13:15:08 +0200." <3BC2DC3C.AEEB3379@inf.ethz.ch> References: <3BC227FC.32750.79A82249@localhost> <3BC2DC3C.AEEB3379@inf.ethz.ch> Message-ID: <200110091115.HAA16907@cj20424-a.reston1.va.home.com> > > Does "importing from zip archives" really need a PEP? It's > > already quasi-official in that Jython and MacPython implement > > it. > > Personally, I think so, a PEP would be indeed a good thing, > mostly to fix the details. AFAIK MacPython and Jython > are already different about the details, at least in Jython > the feature is still experimental so there is space for > changes, don't know in general. > > regards, Samule Pedroni. +1. A PEP is the only way to get this to be a standard feature. --Guido van Rossum (home page: http://www.python.org/~guido/) From Greg.Wilson@baltimore.com Tue Oct 9 15:17:31 2001 From: Greg.Wilson@baltimore.com (Greg Wilson) Date: Tue, 9 Oct 2001 10:17:31 -0400 Subject: [Python-Dev] Byte revived! Message-ID: <930BBCA4CEBBD411BE6500508BB3328F432204@nsamcanms1.ca.baltimore.com> Hi, all. It's finally public --- Byte magazine is being revived: http://www.prnewswire.com/cgi-bin/stories.pl?ACCT=104&STORY=/www/story/10-05 -2001/0001585919 The easiest way to make sure that the new Byte is as good as the old is to contribute content. If you are working on something that's technically sweet, or if you think you know enough about a topic to write a good overview, we would love to hear from you. (In particular, I'd *still* like articles about garbage collection in Python, how Stackless works, and a bunch of other things that I've nagged people about over the past year and a half...) Thanks, Greg Wilson gvwilson at ddj.com ----------------------------------------------------------------------------------------------------------------- The information contained in this message is confidential and is intended for the addressee(s) only. If you have received this message in error or there are any problems please notify the originator immediately. The unauthorized use, disclosure, copying or alteration of this message is strictly forbidden. Baltimore Technologies plc will not be liable for direct, special, indirect or consequential damages arising from alteration of the contents of this message by a third party or as a result of any virus being passed on. In addition, certain Marketing collateral may be added from time to time to promote Baltimore Technologies products, services, Global e-Security or appearance at trade shows and conferences. This footnote confirms that this email message has been swept by Baltimore MIMEsweeper for Content Security threats, including computer viruses. From paul@ActiveState.com Tue Oct 9 20:36:55 2001 From: paul@ActiveState.com (Paul Prescod) Date: Tue, 09 Oct 2001 12:36:55 -0700 Subject: [Python-Dev] Fame, Fortune and the Python Conference Message-ID: <3BC351D7.DF81B466@ActiveState.com> Many of you will have submitted papers for the Python conference. Whether you did or didn't, please consider another way to contribute to the conference. You could submit a talks to the Web Services and Protocols track, Zope or Tools tracks. The Web Services and Protocols track is for anyone who is using Python to communicate information between computers in an innovative way. You can get involved with a simple email back to me saying you are interested with a couple of sentences about your area of interest. I will work with you to turn that into an abstract. You do not have to submit a paper or anything else in advance. Just work with me on your abstract and then show up at the conference with a fascinating talk. We already have some well-known speakers signed up so you will be in good company! Here are some areas of interest: * Jabber peer-to-peer protocol * XML-based Web Services protocols (SOAP, XML-RPC) * CORBA Distributed Computing Protocol * Web Services Description Language * Business Integration techniques ("B2B") * Enterprise Application Integration techniques ("EAI") * Proprietary Protocols Just send me a note this week (why not reply right row!) describing your project with Python and protocols or web services and we'll work out whether there is an abstract in there. More info is available here: http://www.python10.org/p10-conferenceEvents.html#webServices Conference info (dates, location,etc.) is here: http://www.python10.org/ Paul Prescod From arigo@ulb.ac.be Wed Oct 10 11:54:46 2001 From: arigo@ulb.ac.be (Armin Rigo) Date: Wed, 10 Oct 2001 12:54:46 +0200 (MET DST) Subject: [Python-Dev] Psyco3 is working Message-ID: <Pine.OSF.3.96.1011010125034.3330A-100000@aster1.ulb.ac.be> Hello everybody, Psyco, the Python Specializing Comiler, is working --- currently only on very simple examples, but it works well. The previous acceleration of 3 to 5 times is peanuts; now I get a factor 100, including the compilation time! If you don't believe me have a look at it: http://homepages.ulb.ac.be/~arigo/psyco/ Promizing-to-add-a-lot-of-comments-in-the-source'ly yours, Armin From mahler@cyc.com Wed Oct 10 14:22:52 2001 From: mahler@cyc.com (Daniel Mahler) Date: Wed, 10 Oct 2001 08:22:52 -0500 Subject: [Python-Dev] Psyco3 is working In-Reply-To: <Pine.OSF.3.96.1011010125034.3330A-100000@aster1.ulb.ac.be> References: <Pine.OSF.3.96.1011010125034.3330A-100000@aster1.ulb.ac.be> Message-ID: <15300.19372.877863.62136@mcallister.cyc.com> g'day y'all, (this is what happens when ozzies live in texas :)) I was wandering how projects like Psyco, Stackless and Parrot affect each other, since they seem to change aspects of the execution model. regards D Armin Rigo writes: > Hello everybody, > > Psyco, the Python Specializing Comiler, is working --- currently only on > very simple examples, but it works well. The previous acceleration of 3 to > 5 times is peanuts; now I get a factor 100, including the compilation > time! > > If you don't believe me have a look at it: > > http://homepages.ulb.ac.be/~arigo/psyco/ > > > Promizing-to-add-a-lot-of-comments-in-the-source'ly yours, > > Armin > > > _______________________________________________ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev From msw@redhat.com Wed Oct 10 18:27:53 2001 From: msw@redhat.com (Matt Wilson) Date: Wed, 10 Oct 2001 13:27:53 -0400 Subject: [Python-Dev] __del__ and subclassed types Message-ID: <20011010132753.F12576@devserv.devel.redhat.com> Python 2.2a4: class myList(type([])): def __del__(self): print "list dealloced" list = myList() del list never prints "list dealloced". Is this supposed to work? Cheers, Matt From mclay@nist.gov Wed Oct 10 18:35:58 2001 From: mclay@nist.gov (Michael McLay) Date: Wed, 10 Oct 2001 13:35:58 -0400 Subject: [Python-Dev] Type checks of instance variables Message-ID: <01101013355800.02052@fermi.eeel.nist.gov> Many applications require constraining the contents of instance variables to specific types. For instance, the attributes of an element in a validating XML Schema implementation would require constraining the types allowed in each of the instance variables of the class implementing the element definition. This testing for type constraints can be accomplished by adding an isinstance test to the __setattr__ of the class that contains the constrained attributes. Prior to Python 2.2 the resulting class definitions would be bloated with many lines of type checking code. The new builtin property class has simplified the task of constraint checking somewhat. There are some improvements that would make it easier and less error prone to selectively restrict the types of instance variables. The following change to the property class adds a test for the instance variable type to the __set__ method. class property2(object): def __init__(self, fget=None, fset=None, doc=None, type=None): self.fget = fget self.fset = fset self._type = type self.__doc__ = doc def __get__(self, inst, type=None): if self.fget is None: raise AttributeError, "this attribute is write-only" return self.fget(inst) def __set__(self, inst, value): if self.fset is None: raise AttributeError, "this attribute is read-only" if not isinstance(value,self._type): raise AttributeError,"attribute not of type '%s'"%\ self._type.__name__ return self.fset(inst, value) The following example illustrates the use of the property2 type checking capability. When the assignment of a value is made to c.b the type checks are performed as desired. Unfortunately this approach is not sufficient to guard against incorrect types being assigned to the instance variables. There are several examples demonstrating the "hidden" _b instance variable being populated with data of the wrong type. ################# demonstration of property2 ############################## from property2 import property2 class unsignedByte(int): """ An unsignedByte is a constrained integer with a range of 0-255 """ def __init__(self,value): if value < 0 or value > 255: raise ValueError, "unsignedByte must be between 0 and 255" self = value class Color(object): def __init__(self,b= 0 ): # the following assignment does not check the type of "_b". # The default value of "0" is not of the proper type for # the instance variable b as defined by the property2 statement. self._b = b __slots__ = ["_b"] def setb(self, value): self._b = value def getb(self): return self._b def messItUp(self): self._b = 2000 b = property2(getb,setb,type=unsignedByte) c = Color() x = unsignedByte(34) c.b = x print "The value of c.b is ",c.b c.messItUp() print "A method can accidentally mess up the variable", c.b c.b = x c._b = 4000 print "you can override the hidden value as well", c.b # Proper assignment to the instance variable does run the type check. # An exception is raised by the following incorrectly typed variable # assignment c.b = 25 In order for the property2 type checking to work well the mechanism needs to forbid access to the "hidden" container for the value of the instance variable. The type checking could also be combined with the new __slots__ mechanism for declaring variable names. In this approach a TypedSlots class defines the type constraints for the slot names. The following example demonstrates the use of the TypedSlots class (a subclassed dictionary) to define the types allowed for each slot in a class definition. class unsignedByte(int): """ An unsignedByte is a constrained integer with a range of 0-255 """ def __init__(self,value): if value < 0 or value > 255: raise ValueError, "unsignedByte must be between 0 and 255" self = value class TypedSlots(dictionary): """A TypedSlot object defines an object that is assigned to a class.__slot__ definition. The TypedSlot object constrains each slot to a specific set of types. The allowed types are associated with the name at the time the slot names are defined. """ def defineSlot(self,name,types): """Add a new slot to the TypedSlots object. All slot definitions can be added as part of the TypedSlots constructor, or they can be added one at a time using the definedSlot method """ self["_" + name] = (name,types) # build an instance of the TypedSlots object by adding the slots # one at a time. t = TypedSlots() t.defineSlot("r",unsignedByte) t.defineSlot("g",unsignedByte) t.defineSlot("b",unsignedByte) t.defineSlot("name",str) # The following class definition uses the TypedSlot instance example # The dictionary type can be used in place of a list of # strings for defining slots because iterating over a dictionary returns # a list of strings (the keys to the dictionary). class Color(object): __slots__ = t # The bytecodes generated by the following setX and getX methods and # assignment of property to X would be automatically generated when # an object of type TypedSlots is detected as the type for __slots__. def setb(self,value): if isinstance(value,self.__slots__["_b"][1]): self._b = value else: raise TypeError, "b must be of type unsignedByte" def getb(self): if isinstance(self._b,self.__slots__["_b"][1]): return self._b else: raise TypeError, "b must be set to a value of type unsignedByte" b = property(getb,setb) def setr(self,value): if isinstance(value,self.__slots__["_r"][1]): self._r = value else: raise TypeError, "r must be of type unsignedByte" def getr(self): if isinstance(self._r,self.__slots__["_r"][1]): return self._r else: raise TypeError, "r must be set to a value of type unsignedByte" r = property(getr,setr) def setg(self,value): if isinstance(value,self.__slots__["_g"][1]): self._g = value else: raise TypeError, "g must be of type unsignedByte" def getg(self): if isinstance(self._g,self.__slots__["_g"][1]): return self._g else: raise TypeError, "g must be set to a value of type unsignedByte" g = property(getg,setg) def setname(self,value): if isinstance(value,self.__slots__["_name"][1]): self._name = value else: raise TypeError, "name must be of type str" def getname(self): if isinstance(self._name,self.__slots__["_name"][1]): return self._name else: raise TypeError, "name must be set to a value of type str" name = property(getname,setname) c = Color() c.name = "aqua" print "The color name is", c.name x = unsignedByte(254) c.b= x print "The value of the blue component is", c.b # the following statement will fail because the value is not # of type unsignedByte c.g= 255 This implementation also cannot guard the "hidden" names for the slots from access. This creates a security hole in the type constraint system. This hole complicates any attempts to write optimizations that take advantage of the typed slots. The proposed change to the __slots__ implementation would force type checks to be made on all accesses to the variables defined by TypedSlots. The implementation would automatically generate the bytecode for testing the instance type as defined by the TypeSlots definition. The implementation of the example required the slot to have "hidden" names (i.e. the names _r, _b _g, and _name) in order to implement the type checks on the slots for the instance variables. The proposed TypedSlots mechanism would eliminate the "hidden" name and calculate the address of the slot directly. The addition of this mechanism to the __slots__ definition would add protections against violations of a type constraint system for slots. The mechanism would also reduce the code required to define the constraints. Using the TypedSlots definition, the constrained type of the example would be written as follows: class Color(object): __slots__ =TypedSlots({ "r":unsignedByte, "g":unsignedByte, "b":unsignedByte, "name":str}) The contents of the slot would only be accessible though the exposed instance variable name. For example: c = Color() c.r = unsignedByte(126) An optional default value could be defined during the creation of the slots by making the value a tuple: class Color(object): __slots__ =TypedSlots({ "r":(unsignedByte,unsignedByte(0), "g":(unsignedByte,unsignedByte(0), "b":(unsignedByte,unsignedByte(0), "name":str}) If default values are defined the type test on __get__ could be skipped. From jeremy@zope.com Wed Oct 10 19:16:53 2001 From: jeremy@zope.com (Jeremy Hylton) Date: Wed, 10 Oct 2001 14:16:53 -0400 (EDT) Subject: [Python-Dev] Re: [Python-checkins] CVS: python/dist/src/Lib xmlrpclib.py,1.11,1.12 In-Reply-To: <E15rLim-0006cV-00@usw-pr-cvs1.sourceforge.net> References: <E15rLim-0006cV-00@usw-pr-cvs1.sourceforge.net> Message-ID: <15300.37013.942228.195940@slothrop.digicool.com> >>>>> "SM" == Skip Montanaro <montanaro@users.sourceforge.net> writes: SM> Update of /cvsroot/python/python/dist/src/Lib In directory SM> usw-pr-cvs1:/tmp/cvs-serv25433 SM> Modified Files: SM> xmlrpclib.py SM> Log Message: allow long ints to be marshalled as ints - no check SM> is made to the incoming value, so the programmer will have to SM> catch OverflowError. How does the other end unmarshal it? You shouldn't get overflow errors any more. Oh, well, actually you do still get overflow and value errors from the builtin int(). I wonder if there should be a generic number-from-string conversion function that returns either a long or an int as needed. If an integer literal in your program text can produce a long, why can't builtin int()? Jeremy From skip@pobox.com (Skip Montanaro) Wed Oct 10 20:09:41 2001 From: skip@pobox.com (Skip Montanaro) (Skip Montanaro) Date: Wed, 10 Oct 2001 14:09:41 -0500 Subject: [Python-Dev] Re: [Python-checkins] CVS: python/dist/src/Lib xmlrpclib.py,1.11,1.12 In-Reply-To: <15300.37013.942228.195940@slothrop.digicool.com> References: <E15rLim-0006cV-00@usw-pr-cvs1.sourceforge.net> <15300.37013.942228.195940@slothrop.digicool.com> Message-ID: <15300.40181.548940.510295@beluga.mojam.com> SM> xmlrpclib.py SM> Log Message: allow long ints to be marshalled as ints - no check SM> is made to the incoming value, so the programmer will have to SM> catch OverflowError. Jeremy> How does the other end unmarshal it? You shouldn't get overflow Jeremy> errors any more. Jeremy> Oh, well, actually you do still get overflow and value errors Jeremy> from the builtin int(). The receiving end will unmarshal the object into an int. This is no worse than the way xmlrpclib treats tuples. XML-RPC only has an <array> type, which both lists and tuples marshal into. On the receiving end <array> values are always unmarshalled into lists. Jeremy> I wonder if there should be a generic number-from-string Jeremy> conversion function that returns either a long or an int as Jeremy> needed. If an integer literal in your program text can produce Jeremy> a long, why can't builtin int()? Not sure what you're getting at here. You do have to pass in something that smells like an int. "123" would obviously smell like a string and get marshalled as such. The XML-RPC spec also indicates that the <int> tag is restricted to the value that can be represented by a four-byte signed integer. It has no type that would marshal a Python long or a native int that is larger than sys.maxint. Adding that would be difficult to get past Dave Winer on the grounds of interoperability. Skip From jeremy@zope.com Wed Oct 10 20:17:08 2001 From: jeremy@zope.com (Jeremy Hylton) Date: Wed, 10 Oct 2001 15:17:08 -0400 (EDT) Subject: [Python-Dev] Re: [Python-checkins] CVS: python/dist/src/Lib xmlrpclib.py,1.11,1.12 In-Reply-To: <15300.40181.548940.510295@beluga.mojam.com> References: <E15rLim-0006cV-00@usw-pr-cvs1.sourceforge.net> <15300.37013.942228.195940@slothrop.digicool.com> <15300.40181.548940.510295@beluga.mojam.com> Message-ID: <15300.40628.922232.776798@slothrop.digicool.com> >>>>> "SM" == Skip Montanaro <skip@pobox.com> writes: SM> xmlrpclib.py Log Message: allow long ints to be marshalled as SM> ints - no check is made to the incoming value, so the programmer SM> will have to catch OverflowError. Jeremy> How does the other end unmarshal it? You shouldn't get Jeremy> overflow errors any more. Jeremy> Oh, well, actually you do still get overflow and value Jeremy> errors from the builtin int(). SM> The receiving end will unmarshal the object into an int. Sorry to repeat myself but: "Jeremy> How does the other end unmarshal it?" Jeremy> I wonder if there should be a generic number-from-string Jeremy> conversion function that returns either a long or an int as Jeremy> needed. If an integer literal in your program text can Jeremy> produce a long, why can't builtin int()? SM> Not sure what you're getting at here. I meant to say "I wonder if Python should provide a generic string-to-number conversion routine that returns either an int or a long." SM> The XML-RPC spec also SM> indicates that the <int> tag is restricted to the value that can SM> be represented by a four-byte signed integer. It sounds like such a function would not be particularly useful for XML-RPC, but I can imagine it would be useful in a world where programmers don't worry about whether they have an int or a long. Incidentally, it looks like the XML-RPC code won't work with subclasses of built-in types. It's does dispatch on the type() of the object, but a subclass of string won't have type StringType. It seems to me, though, that it should be marshallable using XML-RPC. Jeremy From skip@pobox.com (Skip Montanaro) Wed Oct 10 20:21:21 2001 From: skip@pobox.com (Skip Montanaro) (Skip Montanaro) Date: Wed, 10 Oct 2001 14:21:21 -0500 Subject: [Python-Dev] Re: [pygtk] How to delete Gtk objects In-Reply-To: <20011010132134.E12576@devserv.devel.redhat.com> References: <3BC46024.80B87D73@ijs.si> <20011010130011.C12576@devserv.devel.redhat.com> <20011010132134.E12576@devserv.devel.redhat.com> Message-ID: <15300.40881.489481.1219@beluga.mojam.com> (adding python-dev to this conversation because decisions about missing special methods vis a vis PEP 252/253 seem to me to definitely belong in that space.) >> I think that there may be some problems with extended types and >> __del__? Matt> For example: Matt> class myList(type([])): Matt> def __del__(self): Matt> print "list dealloced" Matt> list = myList() Matt> list.append("foo") Matt> list.append("bar") Matt> del list Matt> never prints list delloced. My guess (without examining any code) is that __del__ is not implemented for new-style classes. There is this rather cryptic note at the end of PEP 253: - open issues: ... - do we need __coerce__, __del__? ... which leads me to believe that __del__ would not be called in the current implementation. If people are going to be subclassing more than the traditional builtin types I think __del__ methods will have to be supported. Skip From paul@ActiveState.com Wed Oct 10 20:33:18 2001 From: paul@ActiveState.com (Paul Prescod) Date: Wed, 10 Oct 2001 12:33:18 -0700 Subject: [Python-Dev] Re: [Python-checkins] CVS: python/dist/src/Lib xmlrpclib.py,1.11,1.12 References: <E15rLim-0006cV-00@usw-pr-cvs1.sourceforge.net> <15300.37013.942228.195940@slothrop.digicool.com> <15300.40181.548940.510295@beluga.mojam.com> <15300.40628.922232.776798@slothrop.digicool.com> Message-ID: <3BC4A27E.1E135B01@ActiveState.com> Jeremy Hylton wrote: > >... > > It sounds like such a function would not be particularly useful for > XML-RPC, but I can imagine it would be useful in a world where > programmers don't worry about whether they have an int or a long. In such a case I would use "long()" and not worry about a slight performance hit. Paul Prescod From skip@pobox.com (Skip Montanaro) Wed Oct 10 20:39:26 2001 From: skip@pobox.com (Skip Montanaro) (Skip Montanaro) Date: Wed, 10 Oct 2001 14:39:26 -0500 Subject: [Python-Dev] Re: [Python-checkins] CVS: python/dist/src/Lib xmlrpclib.py,1.11,1.12 In-Reply-To: <15300.40628.922232.776798@slothrop.digicool.com> References: <E15rLim-0006cV-00@usw-pr-cvs1.sourceforge.net> <15300.37013.942228.195940@slothrop.digicool.com> <15300.40181.548940.510295@beluga.mojam.com> <15300.40628.922232.776798@slothrop.digicool.com> Message-ID: <15300.41966.749814.1125@beluga.mojam.com> Jeremy> I meant to say "I wonder if Python should provide a generic Jeremy> string-to-number conversion routine that returns either an int Jeremy> or a long." SM> The XML-RPC spec also indicates that the <int> tag is restricted to SM> the value that can be represented by a four-byte signed integer. Jeremy> It sounds like such a function would not be particularly useful Jeremy> for XML-RPC, but I can imagine it would be useful in a world Jeremy> where programmers don't worry about whether they have an int or Jeremy> a long. Ah, okay. I was only looking at your note through XML-RPC-colored glasses. I thought PEP 237 was supposed to handle the int/long unification. Presumably, after it is fully in effect, int("9999999999999999999999999") will just return a long, maybe not. Jeremy> Incidentally, it looks like the XML-RPC code won't work with Jeremy> subclasses of built-in types. It's does dispatch on the type() Jeremy> of the object, but a subclass of string won't have type Jeremy> StringType. It seems to me, though, that it should be Jeremy> marshallable using XML-RPC. I filed a bug report just so this thought wouldn't get lost: http://sourceforge.net/tracker/index.php?func=detail&aid=469972&group_id=5470&atid=105470 I don't know if that's a bug report or a feature request though. Xmlrpclib doesn't currently marshal instances of classic classes, so I'm not so sure it should marshal instances of new-style classes either. It's Fredrik's and Martin's call. Skip From jeremy@zope.com Wed Oct 10 20:49:21 2001 From: jeremy@zope.com (Jeremy Hylton) Date: Wed, 10 Oct 2001 15:49:21 -0400 (EDT) Subject: [Python-Dev] Re: [Python-checkins] CVS: python/dist/src/Lib xmlrpclib.py,1.11,1.12 In-Reply-To: <15300.41966.749814.1125@beluga.mojam.com> References: <E15rLim-0006cV-00@usw-pr-cvs1.sourceforge.net> <15300.37013.942228.195940@slothrop.digicool.com> <15300.40181.548940.510295@beluga.mojam.com> <15300.40628.922232.776798@slothrop.digicool.com> <15300.41966.749814.1125@beluga.mojam.com> Message-ID: <15300.42561.319722.570229@slothrop.digicool.com> >>>>> "SM" == Skip Montanaro <skip@pobox.com> writes: Jeremy> It sounds like such a function would not be particularly Jeremy> useful for XML-RPC, but I can imagine it would be useful in Jeremy> a world where programmers don't worry about whether they Jeremy> have an int or a long. SM> Ah, okay. I was only looking at your note through SM> XML-RPC-colored glasses. I thought PEP 237 was supposed to SM> handle the int/long unification. Presumably, after it is fully SM> in effect, int("9999999999999999999999999") will just return a SM> long, maybe not. If that happens, we'll need to fix xmlrpclib so that it doesn't marshal int that don't fit in 4 bytes :-). Jeremy> Incidentally, it looks like the XML-RPC code won't work with Jeremy> subclasses of built-in types. It's does dispatch on the Jeremy> type() of the object, but a subclass of string won't have Jeremy> type StringType. It seems to me, though, that it should be Jeremy> marshallable using XML-RPC. SM> I filed a bug report just so this thought wouldn't get lost: SM> http://sourceforge.net/tracker/index.php?func=detail&aid=469972&group_id=5470&atid=105470 SM> I don't know if that's a bug report or a feature request though. SM> Xmlrpclib doesn't currently marshal instances of classic SM> classes, so I'm not so sure it should marshal instances of SM> new-style classes either. It's Fredrik's and Martin's call. It seems like it ought to handle subclasses of builtin classes like string. Jeremy From martin@loewis.home.cs.tu-berlin.de Wed Oct 10 21:59:09 2001 From: martin@loewis.home.cs.tu-berlin.de (Martin v. Loewis) Date: Wed, 10 Oct 2001 22:59:09 +0200 Subject: [Python-Dev] CVS: python/dist/src/Lib xmlrpclib.py,1.11,1.12 Message-ID: <200110102059.f9AKx9O02319@mira.informatik.hu-berlin.de> > It seems like it ought to handle subclasses of builtin classes like > string. That's desirable, indeed. I think Skip's concern is primarily how to implement that; that is my concern, atleast. I can think of an implementation that replaces def __dump(self, value): try: f = self.dispatch[type(value)] except KeyError: raise TypeError, "cannot marshal %s objects" % type(value) else: f(self, value) with def __dump(self, value): for t in type(value).__mro__: try: f = self.dispatch[t] except KeyError: pass else: f(self, value) return raise TypeError, "cannot marshal %s objects" % type(value) That has several draw-backs, though: - it does not work with Py 1.5, which /F still requires for xmlrpclib, - it may give ambiguous results, in case an object could act as either of the XML-RPC types (it works deterministic, but may still give the undesired outcome) - it doesn't work for classic classes and instances Can you come up with anything better? Regards, Martin P.S. I don't know whether usage of __mro__ itself would be a bug or a feature. I certainly take pride in coming up with that idea :-) From skip@pobox.com (Skip Montanaro) Wed Oct 10 22:17:16 2001 From: skip@pobox.com (Skip Montanaro) (Skip Montanaro) Date: Wed, 10 Oct 2001 16:17:16 -0500 Subject: [Python-Dev] CVS: python/dist/src/Lib xmlrpclib.py,1.11,1.12 In-Reply-To: <200110102059.f9AKx9O02319@mira.informatik.hu-berlin.de> References: <200110102059.f9AKx9O02319@mira.informatik.hu-berlin.de> Message-ID: <15300.47836.277081.93224@beluga.mojam.com> >> It seems like it ought to handle subclasses of builtin classes like >> string. Martin> That's desirable, indeed. I think Skip's concern is primarily Martin> how to implement that; that is my concern, atleast. No, I'm not worried about the implementation. I know one of you wizards can figure that out. I'm worried about the silent loss of information. Given this class definition: >>> class MyString(str): ... def __init__(self, s): ... str.__init__(s) ... self.foo = "xyz" ... >>> s = MyString("sdfsdf") >>> s 'sdfsdf' >>> s.foo 'xyz' Marshalling it into XML-RPC form in the obvious way will only transmit the string "sdfsdf" to the remote end and will lose the object's foo attribute, which might matter to the client and/or the server. XML-RPC does not provide any mechanism for automagically marshalling scalar objects other than ints, booleans, strings, doubles, ISO8601 date/time objects and base64 strings. The only compound objects it will marshal are structs (dicts, and only dicts with strings as keys) and arrays (tuples or lists). If you want to marshal other things, you have to deal with that at the application level. You could subclass xmlrpclib.Marshaller to encode class instances by encoding their __dict__ or whatever __getstate__ returns, but the the marshaller at the other end will have to be aware of your spiffy encoding and recreate an instance of the appropriate class (whatever that might be - remember the remote end just might not be written in Python). Skip From michel@zope.com Wed Oct 10 22:33:15 2001 From: michel@zope.com (Michel Pelletier) Date: Wed, 10 Oct 2001 17:33:15 -0400 Subject: [Python-Dev] __del__ and subclassed types In-Reply-To: <20011010132753.F12576@devserv.devel.redhat.com> Message-ID: <web-2797649@digicool.com> On Wed, 10 Oct 2001 13:27:53 -0400 Matt Wilson <msw@redhat.com> wrote: > Python 2.2a4: > > class myList(type([])): > def __del__(self): > print "list dealloced" > > list = myList() > del list > > never prints "list dealloced". Is this supposed to work? I don't know, but it does work if you *don't* subclass from type([]). But you may already know that. Neither PEP 252 or 253 mention specifics on __del__, so maybe it's a bug? -Michel From andy@reportlab.com Wed Oct 10 22:58:52 2001 From: andy@reportlab.com (Andy Robinson) Date: Wed, 10 Oct 2001 22:58:52 +0100 Subject: [Python-Dev] Type checks of instance variables In-Reply-To: <E15rQkn-0006Ep-00@mail.python.org> Message-ID: <LKENLBBMDHMKBECHIAIACEKDCAAA.andy@reportlab.com> > class Color(object): > __slots__ =TypedSlots({ "r":(unsignedByte,unsignedByte(0), > "g":(unsignedByte,unsignedByte(0), > "b":(unsignedByte,unsignedByte(0), > "name":str}) > This sounds great. In fact, it's almost exactly what we have been trying to do in reportlab/graphics for the last six months: a graphical widget hierarchy which has to check the type of every assignment and maybe trigger recalculations, because it supports a GUI. It was one of the few occasions when I wished I was using Delphi instead of Python :-) Reportlab would certainly be interested in a standardised framework for property-checking. Let me throw in a few more ideas: (1) __slots__ could hold optional attribute doc strings (2) 'one of a list' is a mightily useful type, and we have GUIs that can infer enough from this to build smart editing interfaces. (3) frequently you want the __slots__ for a class to 'inherit' from a parent, which we do by allowing one to pass in the parent's class; it then looks for the __slots__ in that as a starting point. (4) you may also sometimes want to delete things from the __slots__ in a derived class, and to reduce visibility. (5) things can get slow. I'm also wondering if support for type-checking classes could be enhanced by a C extension, so that if one stuck to a basic bunch of types, the penalty of checking every attribute assignment would not be too great. It would still need to call out to Python if the 'checking function' was written in Python. Here's one of our classes: class Marker(Widget): '''A polymorphic class of markers''' _attrMap = AttrMap(BASE=Widget, kind = AttrMapValue( OneOf(None, 'Square', 'Diamond', 'Circle', 'Cross', 'Triangle', 'StarSix', 'Pentagon', 'Hexagon','Heptagon', 'Octagon', 'FilledSquare','FilledCircle', 'FilledDiamond', 'FilledCross','FilledTriangle','FilledStarSix', 'FilledPentagon', 'FilledHexagon', 'FilledHeptagon', 'FilledOctagon', 'Smiley'), desc='marker type name'), size = AttrMapValue(isNumber,desc='marker size'), x = AttrMapValue(isNumber,desc='marker x coordinate'), y = AttrMapValue(isNumber,desc='marker y coordinate'), dx = AttrMapValue(isNumber,desc='marker x coordinate adjustment'), dy = AttrMapValue(isNumber,desc='marker y coordinate adjustment'), angle = AttrMapValue(isNumber,desc='marker rotation'), fillColor = AttrMapValue(isColorOrNone, desc='marker fill colour'), strokeColor = AttrMapValue(isColorOrNone, desc='marker stroke colour'), strokeWidth = AttrMapValue(isNumber, desc='marker stroke width'), ) Regards, Andy Robinson CEO/Chief Architect, Reportlab Inc. From tim@zope.com Wed Oct 10 22:58:45 2001 From: tim@zope.com (Tim Peters) Date: Wed, 10 Oct 2001 17:58:45 -0400 Subject: [Python-Dev] __del__ and subclassed types In-Reply-To: <web-2797649@digicool.com> Message-ID: <BIEJKCLHCIOIHAGOKOLHOEFJCIAA.tim@zope.com> __del__ isn't implemented for new-style classes yet. From guido@python.org Wed Oct 10 23:05:12 2001 From: guido@python.org (Guido van Rossum) Date: Wed, 10 Oct 2001 18:05:12 -0400 Subject: [Python-Dev] Re: [pygtk] How to delete Gtk objects In-Reply-To: Your message of "Wed, 10 Oct 2001 14:21:21 CDT." <15300.40881.489481.1219@beluga.mojam.com> References: <3BC46024.80B87D73@ijs.si> <20011010130011.C12576@devserv.devel.redhat.com> <20011010132134.E12576@devserv.devel.redhat.com> <15300.40881.489481.1219@beluga.mojam.com> Message-ID: <200110102205.f9AM5Ck21449@odiug.digicool.com> > which leads me to believe that __del__ would not be called in the current > implementation. If people are going to be subclassing more than the > traditional builtin types I think __del__ methods will have to be supported. Correct, __del__ is not yet supported. (__coerce__ is, as of 2.2a4 at least). I'm not sure that __del__ should be supported though: it causes problems with the garbage collector. The GC won't destroy objects with a __del__ method because it's impossible to decide in which order to finalize mutually referencing objects. Can somebody (e.g. whoever started this thread) present a good reason why __del__ should be supported? --Guido van Rossum (home page: http://www.python.org/~guido/) From tim@zope.com Wed Oct 10 23:43:16 2001 From: tim@zope.com (Tim Peters) Date: Wed, 10 Oct 2001 18:43:16 -0400 Subject: [Python-Dev] Bug or feature? Unicode vs t# Message-ID: <BIEJKCLHCIOIHAGOKOLHCEFNCIAA.tim@zope.com> Here in current CVS Python: >>> import binascii >>> binascii.hexlify(u'a') '6100' >>> In 2.1.1, it's '61' instead. I don't know what ought to happen, but 2.1.1 suprised me less. binascii inherits its view of the argument via PyArg_ParseTuple's t# code. From paul@ActiveState.com Wed Oct 10 23:53:30 2001 From: paul@ActiveState.com (Paul Prescod) Date: Wed, 10 Oct 2001 15:53:30 -0700 Subject: [Python-Dev] Bug or feature? Unicode vs t# References: <BIEJKCLHCIOIHAGOKOLHCEFNCIAA.tim@zope.com> Message-ID: <3BC4D16A.A6C01C14@ActiveState.com> Tim Peters wrote: > >... > > In 2.1.1, it's '61' instead. I don't know what ought to happen, but 2.1.1 > suprised me less. binascii inherits its view of the argument via > PyArg_ParseTuple's t# code. The documentation for hexlify says: "Return the hexadecimal representation of the binary data." I don't think it should accept a Unicode string as input at all. Paul Prescod From tim@zope.com Thu Oct 11 00:07:39 2001 From: tim@zope.com (Tim Peters) Date: Wed, 10 Oct 2001 19:07:39 -0400 Subject: [Python-Dev] Bug or feature? Unicode vs t# In-Reply-To: <3BC4D16A.A6C01C14@ActiveState.com> Message-ID: <BIEJKCLHCIOIHAGOKOLHEEFPCIAA.tim@zope.com> [Paul Prescod] > The documentation for hexlify says: "Return the hexadecimal > representation of the binary data." I don't think it should accept a > Unicode string as input at all. So change the docs. hexlify() hasn't changed, yet the result has. So long as Unicode objects support the buffer interface, the question is why they've changed how they respond to that interface. This is a question about t#, not hexlify(). From tim.one@home.com Thu Oct 11 03:25:46 2001 From: tim.one@home.com (Tim Peters) Date: Wed, 10 Oct 2001 22:25:46 -0400 Subject: [Python-Dev] Re: [Python-checkins] CVS: python/dist/src/Lib xmlrpclib.py,1.11,1.12 In-Reply-To: <15300.42561.319722.570229@slothrop.digicool.com> Message-ID: <LNBBLJKPBEHFEDALKOLCKEEPMAAA.tim.one@home.com> [Skip] > Presumably, after it is fully in effect, int("9999999999999999999999999") > will just return a long, maybe not. [Jeremy] > If that happens, we'll need to fix xmlrpclib so that it doesn't > marshal int that don't fit in 4 bytes :-). If someone really cares about this, xmlrcplib is already broken in this respect, at least on 64-bit Linux boxes (where Python ints have always been 64 bits). def fits_in_n_bit_signed_int(i, n): signs = i >> (n-1) return signs == 0 or signs == -1 From skip@pobox.com (Skip Montanaro) Thu Oct 11 03:36:06 2001 From: skip@pobox.com (Skip Montanaro) (Skip Montanaro) Date: Wed, 10 Oct 2001 21:36:06 -0500 Subject: [Python-Dev] Re: [Python-checkins] CVS: python/dist/src/Lib xmlrpclib.py,1.11,1.12 In-Reply-To: <LNBBLJKPBEHFEDALKOLCKEEPMAAA.tim.one@home.com> References: <15300.42561.319722.570229@slothrop.digicool.com> <LNBBLJKPBEHFEDALKOLCKEEPMAAA.tim.one@home.com> Message-ID: <15301.1430.548615.437567@beluga.mojam.com> Tim> If someone really cares about this, xmlrcplib is already broken in Tim> this respect, at least on 64-bit Linux boxes (where Python ints Tim> have always been 64 bits). So instead of simply int()ing longs to marshal them, we should do something like val = int(val) if val > 2L**31-1 or val < -2L**31: raise OverflowError ? (Same test needed for ints...) Skip From gward@python.net Thu Oct 11 03:47:36 2001 From: gward@python.net (Greg Ward) Date: Wed, 10 Oct 2001 22:47:36 -0400 Subject: [Python-Dev] Type checks of instance variables In-Reply-To: <LKENLBBMDHMKBECHIAIACEKDCAAA.andy@reportlab.com> References: <E15rQkn-0006Ep-00@mail.python.org> <LKENLBBMDHMKBECHIAIACEKDCAAA.andy@reportlab.com> Message-ID: <20011010224736.A5252@cthulhu.mems-exchange.org> On 10 October 2001, Andy Robinson said: > > class Color(object): > > __slots__ =TypedSlots({ "r":(unsignedByte,unsignedByte(0), > > "g":(unsignedByte,unsignedByte(0), > > "b":(unsignedByte,unsignedByte(0), > > "name":str}) > > > > This sounds great. In fact, it's almost exactly > what we have been trying to do in reportlab/graphics for > the last six months: a graphical widget hierarchy > which has to check the type of every assignment > and maybe trigger recalculations, because it supports > a GUI. It was one of the few occasions when I wished I > was using Delphi instead of Python :-) Reportlab would > certainly be interested in a standardised framework for > property-checking. Allow me to plug Grouch, my attempt at a type-description language and type-enforcement tool for Python. See http://www.mems-exchange.org/software/grouch/ for some mild hype. Grouch works by parsing type descriptions (currently found in class docstrings) and creating an ObjectSchema instance. An object schema consists of: * a collection of atomic types (default: int, long, float, complex, string); all other types in the schema are built from these * a collection of type aliases (eg. "number" could be an alias for "int | float | long", or "FooBar" for "mypkg.foobar.FooBar") * a collection of ClassDefinition objects A class definition, in turn, consists (mainly) of a list of instance attributes and the type of each instance attribute. One of the things you can do with an object schema is use it to walk an object graph, enforcing the attribute types in the class definitions in the schema. The two scripts currently distributed with Grouch do just that: gen_schema parses docstrings, creates an ObjectSchema, and pickles it; check_data loads the pickled schema, and uses it to typecheck some stored object graph (currently either another pickle or a ZODB). The major things missing from Grouch right now are: * simple value constraints (eg. "notnone", "notempty") * fancy value constraints (eg. "oneof") * any knowledge of new-style classes, eg. the __slots__ attribute I would love for some other python-dev'ers to download Grouch and give it a look. In my non-at-all-humble opinion, Grouch is damned neat and provides a lot of cool ideas for anyone interested in deep, rigorous type-checking of Python objects. Greg -- Greg Ward - Unix bigot gward@python.net http://starship.python.net/~gward/ All programmers are playwrights and all computers are lousy actors. From tim.one@home.com Thu Oct 11 03:56:49 2001 From: tim.one@home.com (Tim Peters) Date: Wed, 10 Oct 2001 22:56:49 -0400 Subject: [Python-Dev] Re: [Python-checkins] CVS: python/dist/src/Lib xmlrpclib.py,1.11,1.12 In-Reply-To: <15301.1430.548615.437567@beluga.mojam.com> Message-ID: <LNBBLJKPBEHFEDALKOLCMEFAMAAA.tim.one@home.com> [Skip] > So instead of simply int()ing longs to marshal them, we should do > something like > > val = int(val) > if val > 2L**31-1 or val < -2L**31: > raise OverflowError > > ? (Same test needed for ints...) I don't know -- I'm not an xmlrpc guy. My guess, though, is that we should refuse to produce anything that doesn't meet the letter of the spec (although "the spec" in this case leaves oodles of stuff un-spec-ified, it seemed clear about the limitation to 4 bytes for ints; OTOH, IIRC it didn't specify 2's-complement ints, and if not we can only guess at the intended limits). From skip@pobox.com (Skip Montanaro) Thu Oct 11 04:30:19 2001 From: skip@pobox.com (Skip Montanaro) (Skip Montanaro) Date: Wed, 10 Oct 2001 22:30:19 -0500 Subject: [Python-Dev] Re: [Python-checkins] CVS: python/dist/src/Lib xmlrpclib.py,1.11,1.12 In-Reply-To: <LNBBLJKPBEHFEDALKOLCMEFAMAAA.tim.one@home.com> References: <15301.1430.548615.437567@beluga.mojam.com> <LNBBLJKPBEHFEDALKOLCMEFAMAAA.tim.one@home.com> Message-ID: <15301.4683.666174.785104@beluga.mojam.com> Tim> I don't know -- I'm not an xmlrpc guy. My guess, though, is that Tim> we should refuse to produce anything that doesn't meet the letter Tim> of the spec (although "the spec" in this case leaves oodles of Tim> stuff un-spec-ified, it seemed clear about the limitation to 4 Tim> bytes for ints; OTOH, IIRC it didn't specify 2's-complement ints, Tim> and if not we can only guess at the intended limits). yeah, the spec is pretty casual. As for twos-complement, it's not needed. Everything is ascii... ;-) Skip From mal@lemburg.com Thu Oct 11 09:02:20 2001 From: mal@lemburg.com (M.-A. Lemburg) Date: Thu, 11 Oct 2001 10:02:20 +0200 Subject: [Python-Dev] pickling subclasses of types (Re: [Python-checkins] CVS: python/dist/src/Lib xmlrpclib.py,1.11,1.12) References: <E15rLim-0006cV-00@usw-pr-cvs1.sourceforge.net> <15300.37013.942228.195940@slothrop.digicool.com> <15300.40181.548940.510295@beluga.mojam.com> <15300.40628.922232.776798@slothrop.digicool.com> Message-ID: <3BC5520C.7749FDB@lemburg.com> Jeremy Hylton wrote: > > Incidentally, it looks like the XML-RPC code won't work with > subclasses of built-in types. It's does dispatch on the type() of the > object, but a subclass of string won't have type StringType. It seems to > me, though, that it should be marshallable using XML-RPC. You have a good point there: AFAIK, pickle and marshal both do the same thing. Perhaps at least pickle should be adapted to treat all objects like instances or at least treat subclasses as instances so they make it across to the unpickling end (Guido, perhaps you already have code in place which does this ?!). -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Consulting & Company: http://www.egenix.com/ Python Software: http://www.lemburg.com/python/ From mal@lemburg.com Thu Oct 11 09:13:49 2001 From: mal@lemburg.com (M.-A. Lemburg) Date: Thu, 11 Oct 2001 10:13:49 +0200 Subject: [Python-Dev] Bug or feature? Unicode vs t# References: <BIEJKCLHCIOIHAGOKOLHCEFNCIAA.tim@zope.com> Message-ID: <3BC554BD.A94FE62B@lemburg.com> Tim Peters wrote: > > Here in current CVS Python: > > >>> import binascii > >>> binascii.hexlify(u'a') > '6100' > >>> > > In 2.1.1, it's '61' instead. I don't know what ought to happen, but 2.1.1 > suprised me less. binascii inherits its view of the argument via > PyArg_ParseTuple's t# code. Neither Unicode nor binascii has changed. For some reason the code in Python/getargs.c was changed in a way which removes the distinctin between "t#" and "s#" -- by delegating the actual buffer conversion to convertbuffer() only the bf_getreadbuffer slot gets used for both cases. Whether this is right or wrong is a different topic (binascii.hexlify() should probably work on the bf_getreadbuffer slot anyway since its normal usage is to encode binary data using HEX), but the change in getargs.c will certainly break existing code out there... -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Consulting & Company: http://www.egenix.com/ Python Software: http://www.lemburg.com/python/ From mal@lemburg.com Thu Oct 11 09:28:50 2001 From: mal@lemburg.com (M.-A. Lemburg) Date: Thu, 11 Oct 2001 10:28:50 +0200 Subject: [Python-Dev] Bug or feature? Unicode vs t# References: <BIEJKCLHCIOIHAGOKOLHCEFNCIAA.tim@zope.com> <3BC554BD.A94FE62B@lemburg.com> Message-ID: <3BC55842.5D10E385@lemburg.com> [Changes to getargs.c] The CVS log indicates that Jeremy's refactoring of the code introduced the mentioned change. Jeremy, was this intended or a simple oversight ? -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Consulting & Company: http://www.egenix.com/ Python Software: http://www.lemburg.com/python/ From martin@loewis.home.cs.tu-berlin.de Thu Oct 11 09:25:28 2001 From: martin@loewis.home.cs.tu-berlin.de (Martin v. Loewis) Date: Thu, 11 Oct 2001 10:25:28 +0200 Subject: [Python-Dev] Bug or feature? Unicode vs t# Message-ID: <200110110825.f9B8PSt01106@mira.informatik.hu-berlin.de> > So change the docs. hexlify() hasn't changed, yet the result has. > So long as Unicode objects support the buffer interface, the > question is why they've changed how they respond to that interface. Unicode objects haven't changed in their response to the buffer interface. What has changed is how t# uses this interface. AFAICT, the change has occured in version 2.59 of getargs.c, in response to patch #426072. In 2.1.1, t# used to call the getcharbuffer operation, now it calls the getreadbuffer function. For Unicode objects, these are different: getcharbuffer converts the string using the default encoding into a character string, whereas getreadbuffer returns a pointer to the internal representation (depending on the platform, this makes 2 or 4 bytes per Unicode character, and they may come out either little or big endian). It seems that the problem is in the simplification attempted with said patch. Regards, Martin From guido@python.org Thu Oct 11 10:02:05 2001 From: guido@python.org (Guido van Rossum) Date: Thu, 11 Oct 2001 05:02:05 -0400 Subject: [Python-Dev] pickling subclasses of types (Re: [Python-checkins] CVS: python/dist/src/Lib xmlrpclib.py,1.11,1.12) In-Reply-To: Your message of "Thu, 11 Oct 2001 10:02:20 +0200." <3BC5520C.7749FDB@lemburg.com> References: <E15rLim-0006cV-00@usw-pr-cvs1.sourceforge.net> <15300.37013.942228.195940@slothrop.digicool.com> <15300.40181.548940.510295@beluga.mojam.com> <15300.40628.922232.776798@slothrop.digicool.com> <3BC5520C.7749FDB@lemburg.com> Message-ID: <200110110902.FAA25731@cj20424-a.reston1.va.home.com> > > Incidentally, it looks like the XML-RPC code won't work with > > subclasses of built-in types. It's does dispatch on the type() of the > > object, but a subclass of string won't have type StringType. It seems to > > me, though, that it should be marshallable using XML-RPC. > > You have a good point there: AFAIK, pickle and marshal both > do the same thing. Perhaps at least pickle should be adapted > to treat all objects like instances or at least treat subclasses > as instances so they make it across to the unpickling end (Guido, > perhaps you already have code in place which does this ?!). As of 2.2a4, pickle fully supports subclasses of built-in types. This is done without any changes to pickle or cPickle: the 'object' class has a __reduce__ method that "does the right thing" (it is implemented by calling copy_reg._reduce). The only unsupported feature is __slots__: attributes described by __slots__ aren't saved (this could be fixed by adding more code to copy_reg._reduce). Marshal happens to use a series of PyXxx_Check(obj) tests rather than a switch on obj->ob_type, so it already supports subclasses of built-in types it supports. Note that marshal and pickle make different choices about what to do for instances of subclasses of built-in types: marshal treats these as the corresponding base type, while pickle attempts to save *all* the object's state. I'm not sure what choice XMLRPC should make: it attempts to marshal the state of classic instances as as structures, but these are unmarshalled as dicts, not as instances. It should probably be extended to do the same for instances of subclasses of 'object', but I'm not sure I'm comfortable with it losing some of the object's state in the case of subclasses of built-in types: some state would be lost in either case. For example: class C(str): def __init__(self): self.encoding = "ascii" def set_encoding(self, enc): self.encoding = enc a = C("hello world") a.set_encoding("latin-1") If we choose to marshal 'a' as a string, the encoding is lost; if we choose to marshal it as a record, the data is lost! This actually answers the question for me: *if* a new-style class is a subclass of a built-in type that xmlrpclib supports, its instances should be marshalled as the corresponding base type; otherwise it should be marshalled as a struct. --Guido van Rossum (home page: http://www.python.org/~guido/) From jeremy@zope.com Thu Oct 11 15:05:25 2001 From: jeremy@zope.com (Jeremy Hylton) Date: Thu, 11 Oct 2001 10:05:25 -0400 (EDT) Subject: [Python-Dev] Bug or feature? Unicode vs t# In-Reply-To: <3BC55842.5D10E385@lemburg.com> References: <BIEJKCLHCIOIHAGOKOLHCEFNCIAA.tim@zope.com> <3BC554BD.A94FE62B@lemburg.com> <3BC55842.5D10E385@lemburg.com> Message-ID: <15301.42789.685596.895422@slothrop.digicool.com> >>>>> "MAL" == M -A Lemburg <mal@lemburg.com> writes: MAL> [Changes to getargs.c] The CVS log indicates that Jeremy's MAL> refactoring of the code introduced the mentioned MAL> change. Jeremy, was this intended or a simple oversight ? An oversight. I'll try to set it right. Jeremy From guido@python.org Thu Oct 11 15:08:32 2001 From: guido@python.org (Guido van Rossum) Date: Thu, 11 Oct 2001 10:08:32 -0400 Subject: [Python-Dev] GC head alignment issue Message-ID: <200110111408.KAA26702@cj20424-a.reston1.va.home.com> See this bug report: http://sourceforge.net/tracker/?func=detail&atid=105470&aid=467145&group_id=5470 The GC header adds 12 bytes to the start of the object, but for objects (like complex) that contain a double and require double alignment, that messes up the alignment. Something needs to be done about this: I'm guessing a configuration test for the required alignment of double, and a rounding up of the GC head size to a multiple of that. Should affect only the AS_GC and FROM_GC macros in gcmodule.c and the two uses of sizeof(PyGC_Head) there -- unless you can figure out a way to add a pad field to the GC head only when needed, then a change to objimpl.h might be sufficient. So far this only affects HP hardware, but we can't trust that! (It hasn't affected 2.0 or 2.1 because it only breaks when I *subclass* complex.) --Guido van Rossum (home page: http://www.python.org/~guido/) From jim@interet.com Thu Oct 11 15:48:45 2001 From: jim@interet.com (James C. Ahlstrom) Date: Thu, 11 Oct 2001 10:48:45 -0400 Subject: [Python-Dev] Where is PyMarshal_ReadShortFromFile() ? Message-ID: <3BC5B14D.FC2E673F@interet.com> I am looking at Python 2.2a3. In Include/, marshal.h declares both PyMarshal_ReadLongFromFile() and PyMarshal_ReadShortFromFile(), but the second seems to be missing from marshal.c. Or is it somewhere else? Here is the code; put in marshal.c just before ReadLongFromFile at line 638: int PyMarshal_ReadShortFromFile(FILE *fp) { RFILE rf; rf.fp = fp; return r_short(&rf); } Jim Ahlstrom From guido@python.org Thu Oct 11 16:47:10 2001 From: guido@python.org (Guido van Rossum) Date: Thu, 11 Oct 2001 11:47:10 -0400 Subject: [Python-Dev] Where is PyMarshal_ReadShortFromFile() ? In-Reply-To: Your message of "Thu, 11 Oct 2001 10:48:45 EDT." <3BC5B14D.FC2E673F@interet.com> References: <3BC5B14D.FC2E673F@interet.com> Message-ID: <200110111547.LAA28304@cj20424-a.reston1.va.home.com> > I am looking at Python 2.2a3. > > In Include/, marshal.h declares both PyMarshal_ReadLongFromFile() > and PyMarshal_ReadShortFromFile(), but the second seems to be > missing from marshal.c. Or is it somewhere else? > > Here is the code; put in marshal.c just before ReadLongFromFile at line 638: > > int > PyMarshal_ReadShortFromFile(FILE *fp) > { > RFILE rf; > rf.fp = fp; > return r_short(&rf); > } +1 --Guido van Rossum (home page: http://www.python.org/~guido/) From paul@ActiveState.com Thu Oct 11 17:36:51 2001 From: paul@ActiveState.com (Paul Prescod) Date: Thu, 11 Oct 2001 09:36:51 -0700 Subject: [Python-Dev] Bug or feature? Unicode vs t# References: <200110110825.f9B8PSt01106@mira.informatik.hu-berlin.de> Message-ID: <3BC5CAA3.6A07502F@ActiveState.com> "Martin v. Loewis" wrote: > >... > > For Unicode objects, these are different: getcharbuffer converts the > string using the default encoding into a character string, whereas What is the use case for treating a Unicode object as if it were a buffer of bytes by translating it to the default encoding? "Python objects implemented in C can export a group of functions called the ``buffer interface.'' These functions can be used by an object to expose its data in a raw, byte-oriented format. Clients of the object can use the buffer interface to access the object data directly, without needing to copy it first." Translating the data before handing it to the user violates the "raw"-ness of the byte-oriented format and the principle about not needing to copy it. It behaves quite differently than other implementations of getcharbuffer. Paul Prescod From tim.one@home.com Thu Oct 11 18:04:46 2001 From: tim.one@home.com (Tim Peters) Date: Thu, 11 Oct 2001 13:04:46 -0400 Subject: [Python-Dev] GC head alignment issue In-Reply-To: <200110111408.KAA26702@cj20424-a.reston1.va.home.com> Message-ID: <LNBBLJKPBEHFEDALKOLCMEGNMAAA.tim.one@home.com> [Guido] > ... > So far this only affects HP hardware, but we can't trust that! Doubt it, and the status quo may be inefficient even on HW where "it works" (it's actually common for HW to *allow* for some suboptimal alignments, but at reduced speed). My head hurts, so I'm not sure this is strictly legit C (drop-in replacement for the current PyGC_Head declaration): /* GC information is stored BEFORE the object structure */ typedef union _gc_head { struct { union _gc_head *gc_next; /* not NULL if object is tracked */ union _gc_head *gc_prev; int gc_refs; }; double dummy; /* force worst-case alignment */ } PyGC_Head; Works fine under MSVC-for-Intel, and indeed MSVC-for-Intel prefers (but does not insist upon) 8-byte alignment for doubles. About "strictly legit": I'm not positive it's OK to (A) have an anonymous struct, and then (B) directly reference members of the anonymous struct within a union, e.g. PyGC_Head *p; p->gc_refs = 1; /* guaranteed legit? */ If that's not legit, we could make it legit by naming the struct member and interpolating the name into all existing references. From tim.one@home.com Thu Oct 11 18:11:18 2001 From: tim.one@home.com (Tim Peters) Date: Thu, 11 Oct 2001 13:11:18 -0400 Subject: [Python-Dev] Bug or feature? Unicode vs t# In-Reply-To: <3BC5CAA3.6A07502F@ActiveState.com> Message-ID: <LNBBLJKPBEHFEDALKOLCEEGOMAAA.tim.one@home.com> [Paul Prescod] > What is the use case for treating a Unicode object as if it were a > buffer of bytes by translating it to the default encoding? Obviously, so that you can pass it to binascii.hexlify() <wink>. uselessly y'rs - tim From mclay@nist.gov Thu Oct 11 18:15:09 2001 From: mclay@nist.gov (Michael McLay) Date: Thu, 11 Oct 2001 13:15:09 -0400 Subject: [Python-Dev] Type checks of instance variables Message-ID: <01101113144801.02241@fermi.eeel.nist.gov> On Wednesday 10 October 2001 05:58 pm, Andy Robinson wrote: > Let me throw in a few more ideas: > > (1) __slots__ could hold optional attribute doc strings > (2) 'one of a list' is a mightily useful type, > and we have GUIs that can infer enough from this > to build smart editing interfaces. > (3) frequently you want the __slots__ for a class > to 'inherit' from a parent, which we do by allowing > one to pass in the parent's class; it then looks > for the __slots__ in that as a starting point. > (4) you may also sometimes want to delete things > from the __slots__ in a derived class, and to reduce > visibility. I hope we can use the TypedSlots proposal as a starting point and refine it until it also meets the requirements for ReportLab and Grouch. The following example attempts to add the features you suggested. It extends the original TypedSlots definition to include actions that would be applied to slots. Adding type checking actions would simplify writing a number of applications, such as validation software for XML Schema. The XML Schema type system inspired the Restrict action in the example. The XML Schema type system could be used to identify candidate actions to be added to TypedSlots. class Widget(object): __slots__ = TypedSlots({ "name": CheckType(str, doc="The name by which a widget can be referenced")}) class Color(Widget): __slots__ = TypedSlots({ "r": CheckType(unsignedByte, default = unsignedByte(0)), "g": CheckType(unsignedByte, default = unsignedByte(0)), "b": CheckType(unsignedByte, default = unsignedByte(0)),}) class Shape(Widget): __slots__ = TypedSlots({ "colorRef": CheckType(Color)}) class Circle(Shape): __slots__ = TypedSlots({ "diameter": CheckType(Number,default=1)}) class RedCircle(Circle): """Define a type for red Circles. The Restrict declaration allows the "colorRef" name to be restricted to a subset of the allowed values of the colorRef name. In this case a ReadOnly flag is set to keep the Circle color from being changed and the value is set. """ __slots__ = TypedSlots({"colorRef": Restrict( ReadOnly( Color(r=255)))}) class Drawable(Widget): """The OneOf action allows one of several types to be assigned to the name. Most likely all the types share a common interface. """ __slots__ = TypedSlots({"shapeRef":OneOf(Circle, RedCircle,Rectangle)}) > (5) things can get slow. I'm also wondering if support > for type-checking classes could be enhanced by a C > extension, so that if one stuck to a basic bunch of types, > the penalty of checking every attribute assignment > would not be too great. It would still need to > call out to Python if the 'checking function' was > written in Python. Performance will benefit from making C enhancements. Creating a type verification system that can't be hacked (from Python) will require writing some of the implementation in C. The danger in extending the actions of the TypedSlots to more than a simple type check on __set__ is that it will become to complected to implement in a timely manner. What is the bare minimum required? Specification languages tend to grow to be ugly monsters. Can we start small and add other actions as needed? Could some of the actions be defined in a Python module? > > Here's one of our classes: > > class Marker(Widget): > '''A polymorphic class of markers''' > _attrMap = AttrMap(BASE=Widget, > kind = AttrMapValue( > OneOf(None, 'Square', 'Diamond', > 'Circle', 'Cross', 'Triangle', 'StarSix', > ... My project involves capturing the product information for printed circuit boards and printed circuit assemblies. You r example looks like it came right out of the definition of the graphics required to by tools that draw the layers of a printed circuit board. From mclay@nist.gov Thu Oct 11 18:20:19 2001 From: mclay@nist.gov (Michael McLay) Date: Thu, 11 Oct 2001 13:20:19 -0400 Subject: [Python-Dev] Type checks of instance variables In-Reply-To: <20011010224736.A5252@cthulhu.mems-exchange.org> References: <E15rQkn-0006Ep-00@mail.python.org> <LKENLBBMDHMKBECHIAIACEKDCAAA.andy@reportlab.com> <20011010224736.A5252@cthulhu.mems-exchange.org> Message-ID: <01101113201903.02241@fermi.eeel.nist.gov> On Wednesday 10 October 2001 10:47 pm, Greg Ward wrote: > Grouch works by parsing type descriptions (currently found in class > docstrings) and creating an ObjectSchema instance. An object schema > consists of: Extending the magic of __slots__ was intended to provide a minimally intrusive and pythonic solution to adding type verification to Python. Parsing docstrings to get the type definitions is a pragmatic work-around, but the features of Grouch would benefit by giving them direct language support. > * a collection of atomic types (default: int, long, float, complex, > string); all other types in the schema are built from these > * a collection of type aliases (eg. "number" could be an alias for > "int | float | long", or "FooBar" for "mypkg.foobar.FooBar") > * a collection of ClassDefinition objects > > A class definition, in turn, consists (mainly) of a list of instance > attributes and the type of each instance attribute. That's what the TypedSlots class does, so I think we are on the same page regarding what features are needed. > One of the things you can do with an object schema is use it to walk an > object graph, enforcing the attribute types in the class definitions in > the schema. The two scripts currently distributed with Grouch do just > that: gen_schema parses docstrings, creates an ObjectSchema, and pickles > it; check_data loads the pickled schema, and uses it to typecheck some > stored object graph (currently either another pickle or a ZODB). Are you walking the object graph to enforce the type because it isn't possible to enforce them at creation time in the current Python language? The TypedSlots concept never allows the instance variables to be populated with anything but the allowed types. Adding this concept would have been difficult prior to the addition of __slots__ because it won't require changes that restrict access to the __dict__ of an object. The proposed TypedSlots will require some changes to the C code that implements the __slots__ interpretation. The interpretation of the __slots__ definition would be determined by the type of the __slot__ value. If a list type is detected the interpreter would follow Guido's existing semantics. If the value is a TypedSlots type the interpreter would enforce type checking. The features of the Grouch schema will be helpful in shaping the definition of the type checking actions. > > The major things missing from Grouch right now are: > * simple value constraints (eg. "notnone", "notempty") > * fancy value constraints (eg. "oneof") > * any knowledge of new-style classes, eg. the __slots__ attribute The primary advantage of __slots__ in Python is a reduction in memory usage. The TypedSlots extension is also an efficiency feature. It would simplify the process of accessing the value of slots that are expected to be of a particular type and it may facilitate optimizations based on variable type information. The TypedSlots mechanism would only apply to the variables defined with the new __slots__ mechanism. Would this the constraint be a problem for the types of applications you are building with Grouch? From guido@python.org Thu Oct 11 18:15:49 2001 From: guido@python.org (Guido van Rossum) Date: Thu, 11 Oct 2001 13:15:49 -0400 Subject: [Python-Dev] GC head alignment issue In-Reply-To: Your message of "Thu, 11 Oct 2001 13:04:46 EDT." <LNBBLJKPBEHFEDALKOLCMEGNMAAA.tim.one@home.com> References: <LNBBLJKPBEHFEDALKOLCMEGNMAAA.tim.one@home.com> Message-ID: <200110111715.NAA29098@cj20424-a.reston1.va.home.com> > My head hurts, so I'm not sure this is strictly legit C (drop-in replacement > for the current PyGC_Head declaration): > > /* GC information is stored BEFORE the object structure */ > typedef union _gc_head { > struct { > union _gc_head *gc_next; /* not NULL if object is tracked */ > union _gc_head *gc_prev; > int gc_refs; > }; > double dummy; /* force worst-case alignment */ > } PyGC_Head; Alas, gcc gives me warnings about this: ../Include/objimpl.h:274: warning: unnamed struct/union that defines no instances and references to the fields fail with errors: ../Objects/cellobject.c: In function `PyCell_New': ../Objects/cellobject.c:14: union has no member named `gc_next' [and many more] --Guido van Rossum (home page: http://www.python.org/~guido/) From jeremy@zope.com Thu Oct 11 18:31:08 2001 From: jeremy@zope.com (Jeremy Hylton) Date: Thu, 11 Oct 2001 13:31:08 -0400 (EDT) Subject: [Python-Dev] Bug or feature? Unicode vs t# In-Reply-To: <3BC5CAA3.6A07502F@ActiveState.com> References: <200110110825.f9B8PSt01106@mira.informatik.hu-berlin.de> <3BC5CAA3.6A07502F@ActiveState.com> Message-ID: <15301.55132.722342.122465@slothrop.digicool.com> >>>>> "PP" == Paul Prescod <paul@ActiveState.com> writes: PP> Translating the data before handing it to the user violates the PP> "raw"-ness of the byte-oriented format and the principle about PP> not needing to copy it. It behaves quite differently than other PP> implementations of getcharbuffer. As far as I can tell, all other implementations of getcharbuffer are exact duplicates of getreadbuffer. If the Unicode object doesn't have an appropriate implementation of getcharbuffer, one wonders why the method exists at all: In every case it would be redundant or incorrect. Jeremy From tim.one@home.com Thu Oct 11 18:49:55 2001 From: tim.one@home.com (Tim Peters) Date: Thu, 11 Oct 2001 13:49:55 -0400 Subject: [Python-Dev] GC head alignment issue In-Reply-To: <200110111715.NAA29098@cj20424-a.reston1.va.home.com> Message-ID: <LNBBLJKPBEHFEDALKOLCKEHDMAAA.tim.one@home.com> [Tim] > My head hurts, so I'm not sure this is strictly legit C > (drop-in replacement for the current PyGC_Head declaration): > > /* GC information is stored BEFORE the object structure */ > typedef union _gc_head { > struct { > union _gc_head *gc_next; /* not NULL if object is tracked */ > union _gc_head *gc_prev; > int gc_refs; > }; > double dummy; /* force worst-case alignment */ > } PyGC_Head; [Guido] > Alas, gcc gives me warnings about this: > > ../Include/objimpl.h:274: warning: unnamed struct/union that > defines no instances > > and references to the fields fail with errors: > > ../Objects/cellobject.c: In function `PyCell_New': > ../Objects/cellobject.c:14: union has no member named `gc_next' > [and many more] Then: If that's not legit, we could make it legit by naming the struct member and interpolating the name into all existing references. This is straightforward, if a bit tedious ... OK, I did that. Requires changes to the track and untrack macros, plus a ton of edits inside gcmodule.c. But that's it, and they're all straightforward changes (i.e., low chance of breakage). Should I check it in? From guido@python.org Thu Oct 11 19:01:29 2001 From: guido@python.org (Guido van Rossum) Date: Thu, 11 Oct 2001 14:01:29 -0400 Subject: [Python-Dev] GC head alignment issue In-Reply-To: Your message of "Thu, 11 Oct 2001 13:49:55 EDT." <LNBBLJKPBEHFEDALKOLCKEHDMAAA.tim.one@home.com> References: <LNBBLJKPBEHFEDALKOLCKEHDMAAA.tim.one@home.com> Message-ID: <200110111801.OAA29448@cj20424-a.reston1.va.home.com> > If that's not legit, we could make it legit by naming the struct > member and interpolating the name into all existing references. > > This is straightforward, if a bit tedious ... OK, I did that. Requires > changes to the track and untrack macros, plus a ton of edits inside > gcmodule.c. But that's it, and they're all straightforward changes (i.e., > low chance of breakage). Should I check it in? +1 --Guido van Rossum (home page: http://www.python.org/~guido/) From tim.one@home.com Thu Oct 11 19:35:57 2001 From: tim.one@home.com (Tim Peters) Date: Thu, 11 Oct 2001 14:35:57 -0400 Subject: [Python-Dev] GC head alignment issue In-Reply-To: <200110111801.OAA29448@cj20424-a.reston1.va.home.com> Message-ID: <LNBBLJKPBEHFEDALKOLCOEHFMAAA.tim.one@home.com> [Tim] > If that's not legit, we could make it legit by naming the struct > member and interpolating the name into all existing references. > ... OK, I did that ... Should I check it in? [Guido] > +1 It's checked in. From tim.one@home.com Thu Oct 11 20:15:32 2001 From: tim.one@home.com (Tim Peters) Date: Thu, 11 Oct 2001 15:15:32 -0400 Subject: [Python-Dev] Where is PyMarshal_ReadShortFromFile() ? In-Reply-To: <3BC5B14D.FC2E673F@interet.com> Message-ID: <LNBBLJKPBEHFEDALKOLCKEHJMAAA.tim.one@home.com> [James C. Ahlstrom] > I am looking at Python 2.2a3. > > In Include/, marshal.h declares both PyMarshal_ReadLongFromFile() > and PyMarshal_ReadShortFromFile(), but the second seems to be > missing from marshal.c. Or is it somewhere else? Doesn't appear to be anywhere. > Here is the code; put in marshal.c just before ReadLongFromFile > at line 638: Please submit as a patch to SourceForge, including doc and test case changes. From nas@python.ca Thu Oct 11 20:24:49 2001 From: nas@python.ca (Neil Schemenauer) Date: Thu, 11 Oct 2001 12:24:49 -0700 Subject: [Python-Dev] GC head alignment issue In-Reply-To: <LNBBLJKPBEHFEDALKOLCKEHDMAAA.tim.one@home.com>; from tim.one@home.com on Thu, Oct 11, 2001 at 01:49:55PM -0400 References: <200110111715.NAA29098@cj20424-a.reston1.va.home.com> <LNBBLJKPBEHFEDALKOLCKEHDMAAA.tim.one@home.com> Message-ID: <20011011122449.B6852@glacier.arctrix.com> Tim Peters wrote: > This is straightforward, if a bit tedious ... OK, I did that. Requires > changes to the track and untrack macros, plus a ton of edits inside > gcmodule.c. But that's it, and they're all straightforward changes (i.e., > low chance of breakage). Should I check it in? Sounds good to me. Hopefully you can cover most of it up with macros. Neil From mal@lemburg.com Thu Oct 11 22:08:49 2001 From: mal@lemburg.com (M.-A. Lemburg) Date: Thu, 11 Oct 2001 23:08:49 +0200 Subject: [Python-Dev] Bug or feature? Unicode vs t# References: <200110110825.f9B8PSt01106@mira.informatik.hu-berlin.de> <3BC5CAA3.6A07502F@ActiveState.com> <15301.55132.722342.122465@slothrop.digicool.com> Message-ID: <3BC60A61.21822A13@lemburg.com> Jeremy Hylton wrote: > > >>>>> "PP" == Paul Prescod <paul@ActiveState.com> writes: > > PP> Translating the data before handing it to the user violates the > PP> "raw"-ness of the byte-oriented format and the principle about > PP> not needing to copy it. It behaves quite differently than other > PP> implementations of getcharbuffer. > > As far as I can tell, all other implementations of getcharbuffer are > exact duplicates of getreadbuffer. If the Unicode object doesn't have > an appropriate implementation of getcharbuffer, one wonders why the > method exists at all: In every case it would be redundant or > incorrect. Well, the question is whether hexlify() should use "t#" which is specifically intended to return *character* data and not "s#" which means *binary* data. I think the latter is more appropriate for hexlify() as its intention is to encode binary data. Regarding Paul's question: the getcharbuffer interface was designed with Unicode in mind before a Unicode implementation even existed. It turned out to be a more or less useless design :-( since "character data" is not enough -- you also need to be able to specify an encoding. Unicode only supports this feature because some IO objects rely on it (e.g. the file object when opened in text mode). -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Consulting & Company: http://www.egenix.com/ Python Software: http://www.lemburg.com/python/ From gward@python.net Thu Oct 11 22:12:18 2001 From: gward@python.net (Greg Ward) Date: Thu, 11 Oct 2001 17:12:18 -0400 Subject: [Python-Dev] Type checks of instance variables In-Reply-To: <01101113201903.02241@fermi.eeel.nist.gov> References: <E15rQkn-0006Ep-00@mail.python.org> <LKENLBBMDHMKBECHIAIACEKDCAAA.andy@reportlab.com> <20011010224736.A5252@cthulhu.mems-exchange.org> <01101113201903.02241@fermi.eeel.nist.gov> Message-ID: <20011011171218.A2474@cthulhu.mems-exchange.org> On 11 October 2001, Michael McLay said: > Parsing docstrings to get the type definitions is a pragmatic work-around, > but the features of Grouch would benefit by giving them direct language > support. Yes indeed. > That's what the TypedSlots class does, so I think we are on the same page > regarding what features are needed. I'm sorry, I must have missed something. What is TypedSlots and where is it defined/documented? > Are you walking the object graph to enforce the type because it isn't > possible to enforce them at creation time in the current Python language? More-or-less. I think it's possible to do that with a classic __setattr__, but I avoided going that way out of performance concerns. We run Grouch on our database nightly to detect any type errors that may have crept in in the last 24 hours. That's kinda weird, and specific to applications that work primarily on a large, persistent data store. Using Grouch in another context would require custom coding to query the object schema API for type-correctness of an object or whole object graph. > The TypedSlots concept never allows the instance variables to be populated > with anything but the allowed types. Let me see if I have this straight: __slots__, a built-in feature of new-style classes, catches "self.nubmer = 3" when you meant "self.number = 3". TypedSlots, which I've never heard of before, adds to this by catching "self.number = '3'". Correct? Question: does use of __slots__ guarantee that self.foo is always there? Or do you still have to create it in __init__? > The primary advantage of __slots__ in Python is a reduction in memory > usage. Gee, I would have said the primary advantage is catching "self.nubmer = 3". > The TypedSlots mechanism would only apply to the variables > defined with the new __slots__ mechanism. Would this the constraint be a > problem for the types of applications you are building with Grouch? Not at all. I nearly leapt out of my seat with joy when I saw __slots__ for the first time. In the short term, we (the MEMS Exchange virtual fab, the project for which I wrote Grouch) are utterly dependent on ZODB and ExtensionClass, so won't be able to use new-style classes until ExtensionClass is dead and ZODB is switched to use new-style classes. So Grouch will continue to work as-is for a while. (It already knows about two types of Python classes, namely classic classes and ExtensionClass. Adding a third shouldn't be *too* hard .... >famous last words<) Greg -- Greg Ward - Linux weenie gward@python.net http://starship.python.net/~gward/ If you and a friend are being chased by a lion, it is not necessary to outrun the lion. It is only necessary to outrun your friend. From paulp@ActiveState.com Thu Oct 11 22:15:14 2001 From: paulp@ActiveState.com (Paul Prescod) Date: Thu, 11 Oct 2001 14:15:14 -0700 Subject: [Python-Dev] Bug or feature? Unicode vs t# References: <LNBBLJKPBEHFEDALKOLCEEGOMAAA.tim.one@home.com> Message-ID: <3BC60BE2.D9182D64@ActiveState.com> Tim Peters wrote: > > [Paul Prescod] > > What is the use case for treating a Unicode object as if it were a > > buffer of bytes by translating it to the default encoding? > > Obviously, so that you can pass it to binascii.hexlify() <wink>. This is NOT a question about hexlify. It is a question about t#! Or getcharbuffer. Or something. My real question is whether there is any value in having Unicode objects expose their internal representation to Python programmers through the buffer interface? Paul Prescod From barry@zope.com Thu Oct 11 22:24:40 2001 From: barry@zope.com (Barry A. Warsaw) Date: Thu, 11 Oct 2001 17:24:40 -0400 Subject: [Python-Dev] Bug or feature? Unicode vs t# References: <200110110825.f9B8PSt01106@mira.informatik.hu-berlin.de> <3BC5CAA3.6A07502F@ActiveState.com> <15301.55132.722342.122465@slothrop.digicool.com> <3BC60A61.21822A13@lemburg.com> Message-ID: <15302.3608.196270.320265@anthem.wooz.org> >>>>> "M" == M <mal@lemburg.com> writes: M> Well, the question is whether hexlify() should use "t#" which M> is specifically intended to return *character* data and not M> "s#" which means *binary* data. I think the latter is more M> appropriate for hexlify() as its intention is to encode binary M> data. And I think I agree, even though that /could/ break code. Then again, maybe Paul's suggestion that hexlify() should reject Unicode strings is the better approach. -Barry From guido@python.org Thu Oct 11 22:22:33 2001 From: guido@python.org (Guido van Rossum) Date: Thu, 11 Oct 2001 17:22:33 -0400 Subject: [Python-Dev] Bug or feature? Unicode vs t# In-Reply-To: Your message of "Thu, 11 Oct 2001 14:15:14 PDT." <3BC60BE2.D9182D64@ActiveState.com> References: <LNBBLJKPBEHFEDALKOLCEEGOMAAA.tim.one@home.com> <3BC60BE2.D9182D64@ActiveState.com> Message-ID: <200110112122.RAA00321@cj20424-a.reston1.va.home.com> > My real question is whether there is any value in having Unicode objects > expose their internal representation to Python programmers through the > buffer interface? I used to think so, but I no longer believe this. UTF-16 should be an encoding and that's that. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Thu Oct 11 22:25:42 2001 From: guido@python.org (Guido van Rossum) Date: Thu, 11 Oct 2001 17:25:42 -0400 Subject: [Python-Dev] Bug or feature? Unicode vs t# In-Reply-To: Your message of "Thu, 11 Oct 2001 17:24:40 EDT." <15302.3608.196270.320265@anthem.wooz.org> References: <200110110825.f9B8PSt01106@mira.informatik.hu-berlin.de> <3BC5CAA3.6A07502F@ActiveState.com> <15301.55132.722342.122465@slothrop.digicool.com> <3BC60A61.21822A13@lemburg.com> <15302.3608.196270.320265@anthem.wooz.org> Message-ID: <200110112125.RAA00370@cj20424-a.reston1.va.home.com> > And I think I agree, even though that /could/ break code. Then again, > maybe Paul's suggestion that hexlify() should reject Unicode strings > is the better approach. +1 --Guido van Rossum (home page: http://www.python.org/~guido/) From jim@interet.com Thu Oct 11 22:42:30 2001 From: jim@interet.com (James C. Ahlstrom) Date: Thu, 11 Oct 2001 17:42:30 -0400 Subject: [Python-Dev] Where is PyMarshal_ReadShortFromFile() ? References: <LNBBLJKPBEHFEDALKOLCKEHJMAAA.tim.one@home.com> Message-ID: <3BC61246.6EF6CFA@interet.com> Tim Peters wrote: > Please submit as a patch to SourceForge, including doc and test case > changes. OK, done. But both functions are undocumented and neither is in test_marshal.py. They are internal C utilities. Jim From tim.one@home.com Thu Oct 11 23:04:15 2001 From: tim.one@home.com (Tim Peters) Date: Thu, 11 Oct 2001 18:04:15 -0400 Subject: [Python-Dev] Where is PyMarshal_ReadShortFromFile() ? In-Reply-To: <3BC61246.6EF6CFA@interet.com> Message-ID: <LNBBLJKPBEHFEDALKOLCEEIJMAAA.tim.one@home.com> [Tim] > Please submit as a patch to SourceForge, including doc and test case > changes. [James C. Ahlstrom] > OK, done. Thanks! > But both functions are undocumented and neither is in test_marshal.py. Then you've included the doc and test changes <wink>. > They are internal C utilities. Not so: you're making a bad situation worse. If an extern name doesn't begin with "_", it's part of Python's public C API, and *should* be documented and tested. I don't think anyone will let that hold up your patch, although everyone probably should. including-me-on-both-counts-ly y'rs - tim From fdrake@acm.org Thu Oct 11 23:16:56 2001 From: fdrake@acm.org (Fred L. Drake, Jr.) Date: Thu, 11 Oct 2001 18:16:56 -0400 Subject: [Python-Dev] Where is PyMarshal_ReadShortFromFile() ? In-Reply-To: <LNBBLJKPBEHFEDALKOLCEEIJMAAA.tim.one@home.com> References: <3BC61246.6EF6CFA@interet.com> <LNBBLJKPBEHFEDALKOLCEEIJMAAA.tim.one@home.com> Message-ID: <15302.6744.531834.714184@grendel.zope.com> Tim Peters writes: > documented and tested. I don't think anyone will let that hold up your > patch, although everyone probably should. Of course, it would be easy to write the little bit of LaTeX needed to document the new function. It would have the added benefit that I'd need to document the rest of the PyMarshal_*() functions so the new function would have a place to be listed. Or all sorts of PyMarshal_*() functions could magically grow a leading underscore. ;-) Guido, were all of these really intended to be public, or can the list be trimmed back? They really only appear to be used in import.c and pythonrun.c, with the module init also appear in PC/config.c and Modules/config.c.in (and the generated Modules/config.c). -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> PythonLabs at Zope Corporation From paul@svensson.org Fri Oct 12 00:07:08 2001 From: paul@svensson.org (Paul Svensson) Date: Thu, 11 Oct 2001 19:07:08 -0400 (EDT) Subject: [Python-Dev] GC head alignment issue In-Reply-To: <LNBBLJKPBEHFEDALKOLCKEHDMAAA.tim.one@home.com> Message-ID: <Pine.LNX.4.21.0110111900440.2552-100000@familjen.svensson.org> On Thu, 11 Oct 2001, Tim Peters wrote: >[Tim] >> My head hurts, so I'm not sure this is strictly legit C >> (drop-in replacement for the current PyGC_Head declaration): >> >> /* GC information is stored BEFORE the object structure */ >> typedef union _gc_head { >> struct { >> union _gc_head *gc_next; /* not NULL if object is tracked */ >> union _gc_head *gc_prev; >> int gc_refs; >> }; >> double dummy; /* force worst-case alignment */ >> } PyGC_Head; > >[Guido] >> Alas, gcc gives me warnings about this: >> >> ../Include/objimpl.h:274: warning: unnamed struct/union that >> defines no instances >> >> and references to the fields fail with errors: >> >> ../Objects/cellobject.c: In function `PyCell_New': >> ../Objects/cellobject.c:14: union has no member named `gc_next' >> [and many more] > >Then: > > If that's not legit, we could make it legit by naming the struct > member and interpolating the name into all existing references. > >This is straightforward, if a bit tedious ... OK, I did that. Requires >changes to the track and untrack macros, plus a ton of edits inside >gcmodule.c. But that's it, and they're all straightforward changes (i.e., >low chance of breakage). Should I check it in? The usual thing to do in this situation would be: typedef union _gc_head { struct { union _gc_head *internal_gc_next; union _gc_head *internal_gc_prev; int internal_gc_refs; } gc_internals; double dummy; /* force worst-case alignment */ } PyGC_Head; #define gc_next gc_internals.internal_gc_head #define gc_next gc_internals.internal_gc_head #define gc_refs gc_internals.internal_gc_refs No need to change any other code. /Paul From tim.one@home.com Fri Oct 12 01:54:29 2001 From: tim.one@home.com (Tim Peters) Date: Thu, 11 Oct 2001 20:54:29 -0400 Subject: [Python-Dev] GC head alignment issue In-Reply-To: <Pine.LNX.4.21.0110111900440.2552-100000@familjen.svensson.org> Message-ID: <LNBBLJKPBEHFEDALKOLCGEIPMAAA.tim.one@home.com> [Paul Svensson] > The usual thing to do in this situation would be: > > typedef union _gc_head { > struct { > union _gc_head *internal_gc_next; > union _gc_head *internal_gc_prev; > int internal_gc_refs; > } gc_internals; > double dummy; /* force worst-case alignment */ > } PyGC_Head; > > #define gc_next gc_internals.internal_gc_head > #define gc_next gc_internals.internal_gc_head > #define gc_refs gc_internals.internal_gc_refs Yuck. The less we abuse the preprocessor, the brighter the future of the Python codebase, and that's a good concrete example of why. and-that's-after-ignoring-the-redundant-redefinition-of- gc_next<wink>-ly y'rs - tim From mal@lemburg.com Fri Oct 12 09:19:18 2001 From: mal@lemburg.com (M.-A. Lemburg) Date: Fri, 12 Oct 2001 10:19:18 +0200 Subject: [Python-Dev] Bug or feature? Unicode vs t# References: <LNBBLJKPBEHFEDALKOLCEEGOMAAA.tim.one@home.com> <3BC60BE2.D9182D64@ActiveState.com> <200110112122.RAA00321@cj20424-a.reston1.va.home.com> Message-ID: <3BC6A786.BCB4A4D1@lemburg.com> Guido van Rossum wrote: > > > My real question is whether there is any value in having Unicode objects > > expose their internal representation to Python programmers through the > > buffer interface? > > I used to think so, but I no longer believe this. UTF-16 should be an > encoding and that's that. ... and later ... > > And I think I agree, even though that /could/ break code. Then again, > > maybe Paul's suggestion that hexlify() should reject Unicode strings > > is the better approach. > > +1 Since hexlify() uses a parser marker which does not involve a type check, there's no way to have it reject Unicode objects. BTW, the "s#" parser marker does *not* map to getreadbuffer for Unicode objects. Long ago we decided that the difference between "s#" and "t#" does not make sense for Unicode objects and, in order to increase compatibility of Unicode objects to existing code which uses "s#", to have both parser markers map to getcharbuffer. As a result, both parser markers return the default encoded version of the Unicode object. The getreadbuffer interface is still in place, though... perhaps we ought to consider removing it ?! -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Consulting & Company: http://www.egenix.com/ Python Software: http://www.lemburg.com/python/ From paulp@ActiveState.com Fri Oct 12 09:48:11 2001 From: paulp@ActiveState.com (Paul Prescod) Date: Fri, 12 Oct 2001 01:48:11 -0700 Subject: [Python-Dev] Bug or feature? Unicode vs t# References: <LNBBLJKPBEHFEDALKOLCEEGOMAAA.tim.one@home.com> <3BC60BE2.D9182D64@ActiveState.com> <200110112122.RAA00321@cj20424-a.reston1.va.home.com> <3BC6A786.BCB4A4D1@lemburg.com> Message-ID: <3BC6AE4B.F6B75744@ActiveState.com> "M.-A. Lemburg" wrote: > >... > > Since hexlify() uses a parser marker which does not involve a > type check, there's no way to have it reject Unicode objects. Well, we do have the option of changing the code! We could have hexlify check that its argument type is not Unicode or, more likely, remove the buffer interface from Unicode objects. I think that's the logical outcome of Guido's belief that Python programmers do NOT need access to the internal representation of string objects. The buffer interface seems to be only in existence to give access to internal representations of objects. Paul Prescod From paul@svensson.org Fri Oct 12 13:10:33 2001 From: paul@svensson.org (Paul Svensson) Date: Fri, 12 Oct 2001 08:10:33 -0400 (EDT) Subject: [Python-Dev] GC head alignment issue In-Reply-To: <LNBBLJKPBEHFEDALKOLCGEIPMAAA.tim.one@home.com> Message-ID: <Pine.LNX.4.21.0110120744150.2552-100000@familjen.svensson.org> On Thu, 11 Oct 2001, Tim Peters wrote: >[Paul Svensson] >> The usual thing to do in this situation would be: >> >> typedef union _gc_head { >> struct { >> union _gc_head *internal_gc_next; >> union _gc_head *internal_gc_prev; >> int internal_gc_refs; >> } gc_internals; >> double dummy; /* force worst-case alignment */ >> } PyGC_Head; >> (fixed some typos) >> #define gc_next gc_internals.internal_gc_next >> #define gc_prev gc_internals.internal_gc_prev >> #define gc_refs gc_internals.internal_gc_refs > >Yuck. The less we abuse the preprocessor, the brighter the future of the >Python codebase, and that's a good concrete example of why. Tim, Why do you consider this abuse ? This pattern is used all over the place, you can study it for example in the system headers for linux of glibc. It's perfectly legitimate to not want to spread knowledge of the innards of a struct (it should be wrapped another layer, yes) all over the place. Now, if you claim that for this particular struct, hiding the internals is irrelevant, that's an argument I can buy. I have anuther issue with the change tho: as written, it forces the struct (it's only a union by accident) _gc_head to be aligned, but is it obvious that it really has to pad up the size too ? If for example sizeof (int) is 4, sizeof (struct *) is 8 and sizeof (double) is 16, with 8 byte alignment required for doubles, would a compiler be allowed to make sizeof (struct _gc_head[1]) 24, while still keeping sizeof (struct _gc_head) at 20 ? This would make a difference when making it part of another struct. Try this on for size: -------------------------------------------------------------------------- typedef struct _gc_head { struct _gc_head *internal_gc_next; struct _gc_head *internal_gc_prev; union _gc_head_internals { int _gc_head_internal_gc_refs; double dummy; /* force worst-case alignment */ } } PyGC_Head; #define gc_refs _gc_head_internals._gc_head_internal_gc_refs -------------------------------------------------------------------------- /Paul From mal@lemburg.com Fri Oct 12 12:20:10 2001 From: mal@lemburg.com (M.-A. Lemburg) Date: Fri, 12 Oct 2001 13:20:10 +0200 Subject: [Python-Dev] Bug or feature? Unicode vs t# References: <LNBBLJKPBEHFEDALKOLCEEGOMAAA.tim.one@home.com> <3BC60BE2.D9182D64@ActiveState.com> <200110112122.RAA00321@cj20424-a.reston1.va.home.com> <3BC6A786.BCB4A4D1@lemburg.com> <3BC6AE4B.F6B75744@ActiveState.com> Message-ID: <3BC6D1EA.4201B3C9@lemburg.com> Paul Prescod wrote: > > "M.-A. Lemburg" wrote: > > > >... > > > > Since hexlify() uses a parser marker which does not involve a > > type check, there's no way to have it reject Unicode objects. > > Well, we do have the option of changing the code! > > We could have hexlify check that its argument type is not Unicode or, > more likely, remove the buffer interface from Unicode objects. As mentioned in that email, Unicode objects do not expose their internals via the "s#" or "t#" parser markers. Removing the buffer interface from Unicode would not solve the problem for these. > I think > that's the logical outcome of Guido's belief that Python programmers do > NOT need access to the internal representation of string objects. The > buffer interface seems to be only in existence to give access to > internal representations of objects. I can't follow you here: access to the internals is always possible at C level and even at Python level they can use the unicode-internal codec to peek at the internals. 8-bit strings don't differ between internal and external at all. The question Guido raised really boils down to whether Unicode objects should provide the getreadbuffer interface or not. I don't think it is used in many places since most Python programmers writing C extensions will use the parser markers to get at their arguments. OTOH, getreadbuffer is meant to access the internals of an object, e.g. arrays have this interface too, so this would be an argument for not removing it. -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Consulting & Company: http://www.egenix.com/ Python Software: http://www.lemburg.com/python/ From jim@interet.com Fri Oct 12 14:04:08 2001 From: jim@interet.com (James C. Ahlstrom) Date: Fri, 12 Oct 2001 09:04:08 -0400 Subject: [Python-Dev] Where is PyMarshal_ReadShortFromFile() ? References: <LNBBLJKPBEHFEDALKOLCEEIJMAAA.tim.one@home.com> Message-ID: <3BC6EA48.DB476F16@interet.com> Tim Peters wrote: > Not so: you're making a bad situation worse. If an extern name doesn't > begin with "_", it's part of Python's public C API, and *should* be > documented and tested. I don't think anyone will let that hold up your > patch, although everyone probably should. Yes, I think PyMarshal_ReadLongFromFile() and PyMarshal_ReadShortFromFile() are indeed part of Python's public C API, and that is why they were in marshal.c with public declarations. But they can't be tested by Lib/test_marshal.py because they are not available in the marshal module. I don't know of an existing C API tester, and the names are highly descriptive in lieu of documentation. And marshal.c has been relegated to supporting module.c anyway. In short, I think we are getting a bit carried away. JimA From fdrake@acm.org Fri Oct 12 14:15:03 2001 From: fdrake@acm.org (Fred L. Drake, Jr.) Date: Fri, 12 Oct 2001 09:15:03 -0400 Subject: [Python-Dev] Where is PyMarshal_ReadShortFromFile() ? In-Reply-To: <3BC6EA48.DB476F16@interet.com> References: <LNBBLJKPBEHFEDALKOLCEEIJMAAA.tim.one@home.com> <3BC6EA48.DB476F16@interet.com> Message-ID: <15302.60631.186267.843484@grendel.zope.com> James C. Ahlstrom writes: > Yes, I think PyMarshal_ReadLongFromFile() and PyMarshal_ReadShortFromFile() > are indeed part of Python's public C API, and that is why they were in > marshal.c with public declarations. I think my "should they be?" question remains valid. > But they can't be tested by Lib/test_marshal.py because they are not > available in the marshal module. I don't know of an existing C API > tester, and the names are highly descriptive in lieu of documentation. Modules/_testcapimodule.c. The long names we use for the C API make reading code that uses the API very comfortable, but doesn't help with discoverability. The reference documentation could use some improvements for that, but generally does much better than the headers, if only because you don't need to be familiar with the arrangement of all the different headers. (API users should not need to be familiar with all the headers; they should only need to know to #include Python.h.) > And marshal.c has been relegated to supporting module.c anyway. > In short, I think we are getting a bit carried away. Is this a comment on the testing or the documentation requirements? If the function is being sufficiently covered by other tests, then we don't need to add new tests. Since it was missing, I have to presume it wasn't being called. That tells me that it either isn't needed at all or some test needs to exercise it. As for the documentation, I think the discoverability issue I brought up indicates that we're doing the right thing -- its a lot easier to learn the public API with documentation than without it. I don't know about the rest of the PythonLabs group, but I certainly find the C API documentaiton useful. -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> PythonLabs at Zope Corporation From jim@interet.com Fri Oct 12 14:45:39 2001 From: jim@interet.com (James C. Ahlstrom) Date: Fri, 12 Oct 2001 09:45:39 -0400 Subject: [Python-Dev] Where is PyMarshal_ReadShortFromFile() ? References: <LNBBLJKPBEHFEDALKOLCEEIJMAAA.tim.one@home.com> <3BC6EA48.DB476F16@interet.com> <15302.60631.186267.843484@grendel.zope.com> Message-ID: <3BC6F403.FE3A2454@interet.com> "Fred L. Drake, Jr." wrote: > > James C. Ahlstrom writes: > > Yes, I think PyMarshal_ReadLongFromFile() and PyMarshal_ReadShortFromFile() > > are indeed part of Python's public C API, and that is why they were in > > marshal.c with public declarations. > > I think my "should they be?" question remains valid. Ooops, I meant "they were in marshal.h". Yes they should be public, because PyMarshal_ReadLongFromFile() is used in import.c to read the Python byte code signature MAGIC from a *.pyc file. I need a PyMarshal_ReadShortFromFile() to read zip archives in import.c. > Modules/_testcapimodule.c. Thanks, I knew it had to be somewhere but I couldn't find it. > If the function is being sufficiently covered by other tests, then we > don't need to add new tests. Since it was missing, I have to presume > it wasn't being called. That tells me that it either isn't needed at > all or some test needs to exercise it. Not true. PyMarshal_ReadLongFromFile() is just a wraper function for the function r_long() which is used internally by marshal. Similarly, there is already an r_short() used internally by marshal. Presumably, marshal would't work if these fail, so I assume they work. The PyMarshal_ReadShortFromFile() is just a wraper for r_short(), not new funtionality. > As for the documentation, I think the discoverability issue I > brought up indicates that we're doing the right thing -- its a lot > easier to learn the public API with documentation than without it. I > don't know about the rest of the PythonLabs group, but I certainly > find the C API documentaiton useful. Yes, I find it essential, and wish there were more. JimA From nas@python.ca Fri Oct 12 15:08:07 2001 From: nas@python.ca (Neil Schemenauer) Date: Fri, 12 Oct 2001 07:08:07 -0700 Subject: [Python-Dev] GC head alignment issue In-Reply-To: <Pine.LNX.4.21.0110120744150.2552-100000@familjen.svensson.org>; from paul@svensson.org on Fri, Oct 12, 2001 at 08:10:33AM -0400 References: <LNBBLJKPBEHFEDALKOLCGEIPMAAA.tim.one@home.com> <Pine.LNX.4.21.0110120744150.2552-100000@familjen.svensson.org> Message-ID: <20011012070807.A8457@glacier.arctrix.com> Paul Svensson wrote: > On Thu, 11 Oct 2001, Tim Peters wrote: > >[Paul Svensson] > >> #define gc_next gc_internals.internal_gc_next > >Yuck. > Why do you consider this abuse ? This pattern is used all over the place, > you can study it for example in the system headers for linux of glibc. > It's perfectly legitimate to not want to spread knowledge of the innards > of a struct (it should be wrapped another layer, yes) all over the place. Huh? Say I include a header that abomination in it. Now consider the following line of code: PyGC_Head *gc_next; Oops. The only reasonable alternative to Tim's approach that I can think of would be: #define GC_NEXT(op) ((AS_GC(op))->gc.gc_next) #define GC_PREV(op) ((AS_GC(op))->gc.gc_prev) #define GC_REFS(op) ((AS_GC(op))->gc.gc_refs) I don't think that's any better. Neil From guido@python.org Fri Oct 12 15:54:48 2001 From: guido@python.org (Guido van Rossum) Date: Fri, 12 Oct 2001 10:54:48 -0400 Subject: [Python-Dev] Where is PyMarshal_ReadShortFromFile() ? In-Reply-To: Your message of "Thu, 11 Oct 2001 18:16:56 EDT." <15302.6744.531834.714184@grendel.zope.com> References: <3BC61246.6EF6CFA@interet.com> <LNBBLJKPBEHFEDALKOLCEEIJMAAA.tim.one@home.com> <15302.6744.531834.714184@grendel.zope.com> Message-ID: <200110121454.KAA04951@cj20424-a.reston1.va.home.com> > Of course, it would be easy to write the little bit of LaTeX needed > to document the new function. It would have the added benefit that > I'd need to document the rest of the PyMarshal_*() functions so the > new function would have a place to be listed. > Or all sorts of PyMarshal_*() functions could magically grow a > leading underscore. ;-) > Guido, were all of these really intended to be public, or can the > list be trimmed back? They really only appear to be used in import.c > and pythonrun.c, with the module init also appear in PC/config.c and > Modules/config.c.in (and the generated Modules/config.c). Hm. I consider them pretty internal, but on the other hand there are uses for these outside the core (e.g. a hypothetical extension that might want to manipulate .pyc files). So let's document them. --Guido van Rossum (home page: http://www.python.org/~guido/) From jim@interet.com Fri Oct 12 16:41:09 2001 From: jim@interet.com (James C. Ahlstrom) Date: Fri, 12 Oct 2001 11:41:09 -0400 Subject: [Python-Dev] Where is PyMarshal_ReadShortFromFile() ? References: <LNBBLJKPBEHFEDALKOLCEEIJMAAA.tim.one@home.com> <3BC6EA48.DB476F16@interet.com> <15302.60631.186267.843484@grendel.zope.com> Message-ID: <3BC70F15.206D6C24@interet.com> "Fred L. Drake, Jr." wrote: > don't know about the rest of the PythonLabs group, but I certainly > find the C API documentaiton useful. Hmmm. It occurs to me that I should write some. I just submitted another patch with the docs. JimA From fdrake@acm.org Fri Oct 12 16:53:45 2001 From: fdrake@acm.org (Fred L. Drake, Jr.) Date: Fri, 12 Oct 2001 11:53:45 -0400 Subject: [Python-Dev] Where is PyMarshal_ReadShortFromFile() ? In-Reply-To: <3BC70F15.206D6C24@interet.com> References: <LNBBLJKPBEHFEDALKOLCEEIJMAAA.tim.one@home.com> <3BC6EA48.DB476F16@interet.com> <15302.60631.186267.843484@grendel.zope.com> <3BC70F15.206D6C24@interet.com> Message-ID: <15303.4617.550167.403762@grendel.zope.com> James C. Ahlstrom writes: > Hmmm. It occurs to me that I should write some. I just > submitted another patch with the docs. Thanks! -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> PythonLabs at Zope Corporation From guido@python.org Fri Oct 12 20:06:12 2001 From: guido@python.org (Guido van Rossum) Date: Fri, 12 Oct 2001 15:06:12 -0400 Subject: [Python-Dev] GC head alignment issue In-Reply-To: Your message of "Fri, 12 Oct 2001 08:10:33 EDT." <Pine.LNX.4.21.0110120744150.2552-100000@familjen.svensson.org> References: <Pine.LNX.4.21.0110120744150.2552-100000@familjen.svensson.org> Message-ID: <200110121906.PAA13531@cj20424-a.reston1.va.home.com> > I have anuther issue with the change tho: as written, it forces > the struct (it's only a union by accident) _gc_head to be aligned, > but is it obvious that it really has to pad up the size too ? Yes. > If for example sizeof (int) is 4, sizeof (struct *) is 8 and > sizeof (double) is 16, with 8 byte alignment required for doubles, > would a compiler be allowed to make sizeof (struct _gc_head[1]) 24, > while still keeping sizeof (struct _gc_head) at 20 ? No. Each item in an array has to be aligned, so the sizeof a struct must be a multiple of the largest required alignment for any of its items. It's all spelled out in the C standard, so don't worry. --Guido van Rossum (home page: http://www.python.org/~guido/) From fdrake@acm.org Fri Oct 12 20:15:53 2001 From: fdrake@acm.org (Fred L. Drake, Jr.) Date: Fri, 12 Oct 2001 15:15:53 -0400 Subject: [Python-Dev] profiler hooks Message-ID: <15303.16745.570827.948656@grendel.zope.com> For those who aren't already aware of it, I've been working on a new profiler for Python, implemented in C. It includes features that will allow per-line profiling if desired. I should have the implementation checked in shortly. One thing that I'd like to change is the way the profiler/tracer are dispatched from the main eval loop. The changes are easy enough, but I can simplify things further if we don't need to support both a profiler and tracer simultaneously. (The trace hook is used by sys.settrace(), needed for the debugger and 3rd-party coverage tools; the profile hook is used for (you guessed it!) the profiler, via sys.setprofile().) Would anyone object to working things so that only one of the two can be installed at a time? They can both operate, but it doesn't make sense to use the trace hook while a profiler is active. -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> PythonLabs at Zope Corporation From skip@pobox.com (Skip Montanaro) Fri Oct 12 20:46:56 2001 From: skip@pobox.com (Skip Montanaro) (Skip Montanaro) Date: Fri, 12 Oct 2001 14:46:56 -0500 Subject: [Python-Dev] profiler hooks In-Reply-To: <15303.16745.570827.948656@grendel.zope.com> References: <15303.16745.570827.948656@grendel.zope.com> Message-ID: <15303.18608.490130.307506@beluga.mojam.com> Fred> ... I can simplify things further if we don't need to support both Fred> a profiler and tracer simultaneously. Would anyone object to Fred> working things so that only one of the two can be installed at a Fred> time? +1 Skip From pedroni@inf.ethz.ch Fri Oct 12 21:16:16 2001 From: pedroni@inf.ethz.ch (Samuele Pedroni) Date: Fri, 12 Oct 2001 22:16:16 +0200 Subject: [Python-Dev] profiler hooks References: <15303.16745.570827.948656@grendel.zope.com> Message-ID: <007401c1535a$c0977d00$8e0bfea9@newmexico> > > For those who aren't already aware of it, I've been working on a new > profiler for Python, implemented in C. It includes features that will > allow per-line profiling if desired. I should have the implementation > checked in shortly. > One thing that I'd like to change is the way the profiler/tracer are > dispatched from the main eval loop. The changes are easy enough, but > I can simplify things further if we don't need to support both a > profiler and tracer simultaneously. (The trace hook is used by > sys.settrace(), needed for the debugger and 3rd-party coverage tools; > the profile hook is used for (you guessed it!) the profiler, via > sys.setprofile().) Will your changes propagate to changes to pdb etc? because in that case they will need to be ported to jython, that actually supports both tracers (pdb ) and profilers (profile ). regards. From fdrake@acm.org Fri Oct 12 22:03:36 2001 From: fdrake@acm.org (Fred L. Drake, Jr.) Date: Fri, 12 Oct 2001 17:03:36 -0400 Subject: [Python-Dev] profiler hooks In-Reply-To: <007401c1535a$c0977d00$8e0bfea9@newmexico> References: <15303.16745.570827.948656@grendel.zope.com> <007401c1535a$c0977d00$8e0bfea9@newmexico> Message-ID: <15303.23208.240933.689397@grendel.zope.com> Samuele Pedroni writes: > Will your changes propagate to changes to pdb etc? > because in that case they will need to be ported > to jython, that actually supports both tracers (pdb ) and profilers > (profile ). Changes to bdb/pdb will probably be required, but I'm not sure yet. The biggest change I expect for debugging is that a 'return' event will be generated even when an exception is being propogated, but what this means is that the code that runs for the 'exception' event will not need to be so magical (this was why the old profiler results were so terribly difficult to interpret -- the times were often wrong depending on just how exceptions were handled in the application code). The recent changes to ceval.c for the profiler ensure that the profiler receives a 'return' event when control leaves a frame, even if an exception was set, and expception events are no longer reported to the profiler. This avoids the messy code needed to ensure the profiler accounted for timimg properly for 'exception' events, which is still quite fragile -- it works very hard just to make sure it doesn't lose track of the stack. Making the tracer also receive matching call/return events means that the exception events should no longer need to worry about stack management, which should make the code simpler. I don't know if Jython uses the same pdb.py as CPython, so I don't know what affects will show up there. -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> PythonLabs at Zope Corporation From pedroni@inf.ethz.ch Fri Oct 12 22:19:14 2001 From: pedroni@inf.ethz.ch (Samuele Pedroni) Date: Fri, 12 Oct 2001 23:19:14 +0200 Subject: [Python-Dev] profiler hooks References: <15303.16745.570827.948656@grendel.zope.com><007401c1535a$c0977d00$8e0bfea9@newmexico> <15303.23208.240933.689397@grendel.zope.com> Message-ID: <004101c15363$8c9dbce0$8e0bfea9@newmexico> > > Changes to bdb/pdb will probably be required, but I'm not sure yet. > The biggest change I expect for debugging is that a 'return' event > will be generated even when an exception is being propogated, but what > this means is that the code that runs for the 'exception' event will > not need to be so magical (this was why the old profiler results were > so terribly difficult to interpret -- the times were often wrong > depending on just how exceptions were handled in the application > code). > The recent changes to ceval.c for the profiler ensure that the > profiler receives a 'return' event when control leaves a frame, even > if an exception was set, and expception events are no longer reported > to the profiler. This avoids the messy code needed to ensure the > profiler accounted for timimg properly for 'exception' events, which > is still quite fragile -- it works very hard just to make sure it > doesn't lose track of the stack. Making the tracer also receive > matching call/return events means that the exception events should no > longer need to worry about stack management, which should make the > code simpler. > I don't know if Jython uses the same pdb.py as CPython, so I don't > know what affects will show up there. > I know, and yes pdb.py worked with jython too, that's why I asked about the changes in the first place ... another to-do for jython 2.2 regards. From tim@zope.com Fri Oct 12 23:55:40 2001 From: tim@zope.com (Tim Peters) Date: Fri, 12 Oct 2001 18:55:40 -0400 Subject: [Python-Dev] GC head alignment issue In-Reply-To: <Pine.LNX.4.21.0110120744150.2552-100000@familjen.svensson.org> Message-ID: <BIEJKCLHCIOIHAGOKOLHCEIICIAA.tim@zope.com> [Paul Svensson] > ... > #define gc_next gc_internals.internal_gc_next > ... > Why do you consider this abuse ? Such macros would be visible to every file #include'ing Python.h, and it's certainly not our intent to remove 3 ordinary identifiers from the namespace of every program extending or embedding Python. That brittleness makes it "abuse" on its own. Short of that, it complicates life for developers more than the straightforward alternate I checked in. For example, ask a random debugger for the current value of p->gc_next, and chances are great it will complain that there is no such member (not doing macro expansion itself); or, on the flip side, ask for a dump of a PyGC_Head*, and you'll see fields with names that are utterly surprising because never mentioned in the code using PyGC_Head pointers. Telling and using the plain truth doesn't cause any difficulties, save a tiny bit of extra typing I was happy to do (if I didn't know better, I'd think *you* had to insert "gc." a hundred times yourself <wink>). > This pattern is used all over the place, you can study it for example in > the system headers for linux of glibc. Perhaps one reason Python is more stable than, and spends thousands of lines worming around bugs in, the C libraries it relies on <wink>. > It's perfectly legitimate to not want to spread knowledge of the innards > of a struct (it should be wrapped another layer, yes) all over the place. It's not all over the place -- only the track and untrack macros, and the gc module, have any business mucking with this union (nee struct); indeed, it's in the header file only so that the macros can get at the guts. > ... > I have anuther issue with the change tho: as written, it forces > the struct (it's only a union by accident) _gc_head to be aligned, > but is it obvious that it really has to pad up the size too ? It may not have to, and that's "a feature". > If for example sizeof (int) is 4, sizeof (struct *) is 8 and > sizeof (double) is 16, with 8 byte alignment required for doubles, > would a compiler be allowed to make sizeof (struct _gc_head[1]) 24, > while still keeping sizeof (struct _gc_head) at 20 ? A std-conforming C compiler is not allowed to do that, as, in general, the compiler can't know whether a struct S* points to the first S in an array of S, or to an "isolated" S. So all Ss must have the same layout, regardless of context. As Guido said, the C std goes on at length about this stuff. > This would make a difference when making it part of another struct. Again, the compiler generally can't know whether a struct S* points to an instance of S embedded in another struct either, so again it can't change layout depending on context. > Try this on for size: > ---------------------------------------------------------------------- > typedef struct _gc_head { > struct _gc_head *internal_gc_next; > struct _gc_head *internal_gc_prev; > union _gc_head_internals { > int _gc_head_internal_gc_refs; > double dummy; /* force worst-case alignment */ > } > } PyGC_Head; > > #define gc_refs _gc_head_internals._gc_head_internal_gc_refs > ------------------------------------------------------------------------- As before, I've got no use for the macro. Given that, I'd rather be consistent in sticking a "gc." prefix on every struct member than on just one of them. From tim.one@home.com Sat Oct 13 04:56:58 2001 From: tim.one@home.com (Tim Peters) Date: Fri, 12 Oct 2001 23:56:58 -0400 Subject: [Python-Dev] Where is PyMarshal_ReadShortFromFile() ? In-Reply-To: <3BC6F403.FE3A2454@interet.com> Message-ID: <LNBBLJKPBEHFEDALKOLCKELMMAAA.tim.one@home.com> [James C. Ahlstrom] > Yes they should be public, because PyMarshal_ReadLongFromFile() is used > in import.c to read the Python byte code signature MAGIC from a *.pyc > file. I need a PyMarshal_ReadShortFromFile() to read zip archives in > import.c. There's a finer distinction to be drawn: "public" doesn't mean "C extern", it means a conscious decision that if the API in question ever changes, we promise to bump the Python API version number and so force extension modules to be recompiled (or at least irritate the user with warnings that they *should* be recompiled). IOW, "public" means "anybody in the world can use this, in or out of the core distribution, and we promise it will never break silently across releases". If you use a C extern function like (say) _PyString_FormatLong outside of the core, and we (say) change the calling sequence, or even get rid of it, tough luck -- we don't promise anything about functions (or macros) whose names begin with an underscore (they're part of "the private C API", C extern or not). import.c is part of the core, so can use all the private API functions it likes (and we're responsible for making sure it works across releases no matter how wildly the private API functions change). From neal@metaslash.com Sat Oct 13 16:30:56 2001 From: neal@metaslash.com (Neal Norwitz) Date: Sat, 13 Oct 2001 11:30:56 -0400 Subject: [Python-Dev] pychecker warnings from python-current Message-ID: <3BC85E30.FACC6C19@metaslash.com> Here's a new set of warnings from pychecker (current) against python-current as of 13 Oct: cgi.py:250: No global (ib) found (no idea which variable is desired, best guess is boundary) gzip.py:292: No global (f) found (f should be self?) markupbase.py:306: Invalid arguments to (error), got 2, expected 1 (not sure if this is a problem or not) rexec.py:262: Using the return value from (execfile) which is always None (not really a problem, but implies there is a return value other than None) smtplib.py:540: No global (SMTPError) found (should be SMTPException?) urllib.py:587: Invalid arguments to (http_error_default), got 5, expected 6 (needs errcode passed as 4th arg, between fp & errmsg) webbrowser.py:182: No global (_os) found (should be os) There's also a problem in Objects/stringobject.c on line 2541. The docstring for ''.splitlines() has an extra ]. Neal From fdrake@acm.org Sat Oct 13 16:54:37 2001 From: fdrake@acm.org (Fred L. Drake, Jr.) Date: Sat, 13 Oct 2001 11:54:37 -0400 Subject: [Python-Dev] pychecker warnings from python-current In-Reply-To: <3BC85E30.FACC6C19@metaslash.com> References: <3BC85E30.FACC6C19@metaslash.com> Message-ID: <15304.25533.445048.365261@grendel.zope.com> Neal Norwitz writes: > markupbase.py:306: Invalid arguments to (error), got 2, expected 1 > (not sure if this is a problem or not) ... > webbrowser.py:182: No global (_os) found > (should be os) > > There's also a problem in Objects/stringobject.c on line 2541. The docstring > for ''.splitlines() has an extra ]. These have been fixed; we'll have to look at the rest later. (I have to take a child to a soccor game now!) Thanks! -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> PythonLabs at Zope Corporation From fdrake@acm.org Sat Oct 13 19:33:38 2001 From: fdrake@acm.org (Fred L. Drake, Jr.) Date: Sat, 13 Oct 2001 14:33:38 -0400 Subject: [Python-Dev] pychecker warnings from python-current In-Reply-To: <3BC85E30.FACC6C19@metaslash.com> References: <3BC85E30.FACC6C19@metaslash.com> Message-ID: <15304.35074.718737.112730@grendel.zope.com> Neal Norwitz writes: > Here's a new set of warnings from pychecker (current) against > python-current as of 13 Oct: Ok, I'm back from soccer. ;-) Your suggested changes all look good to me; I've checked them in. Now we just need test cases for them! -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> PythonLabs at Zope Corporation From Keila - Curitiba - Pr"<pastorinha@ieg.com.br Sun Oct 14 03:13:12 2001 From: Keila - Curitiba - Pr"<pastorinha@ieg.com.br (Keila - Curitiba - Pr) Date: Sat, 13 Oct 2001 23:13:12 -0300 Subject: [Python-Dev] UM SITE DE PROTESTO Message-ID: <E15saoy-0002f0-00@mail.python.org> Olá! Veja meu site pessoal no "Tripod.com.br". Basta clicar no endereço abaixo. GARANTO SER SUI-GENERIS - CLIQUE ABAIXO: http://pastorinha.tripod.com.br/seminarista Mais de 61.000 internautas visitaram a PG., existe 7 Álbuns: Se você quiser, por favor, indique minha Home Page, a outros Internautas. Mais detalhes, se comunique, passe um e-mail, que responderei brevemente. Dentro da Home Page, ao lado das fotos, você poderá saber muito mais sobre mim! Obrigada. e-mail: pastorinha@ieg.com.br Beijos:- Keila - Curitiba - Pr - Podes falar comigo, direto dela. Brevemente uma Carta Aberta. http://pastorinha.tripod.com.br/seminarista "Esta mensagem é enviada com a complacência da nova legislação sobre correio eletrônico, Seção 301, Parágrafo (a) (2) (c) Decreto S. 1618, Título Terceiro aprovado pelo "105º Congresso Base das Normativas Internacionais sobre o SPAM". Este E-mail não poderá ser considerado SPAM quando incluir uma forma de ser removido. Para ser removido de futuros correios, simplesmente responda indicando no Assunto: REMOVER" From fdrake@acm.org Sun Oct 14 05:46:12 2001 From: fdrake@acm.org (Fred L. Drake) Date: Sun, 14 Oct 2001 00:46:12 -0400 (EDT) Subject: [Python-Dev] [development doc updates] Message-ID: <20011014044612.90DE228697@beowolf.digicool.com> The development version of the documentation has been updated: http://python.sourceforge.net/devel-docs/ Added documentation for the PyMarshal_*() functions in the Python/C API. From tim.one@home.com Sun Oct 14 20:50:36 2001 From: tim.one@home.com (Tim Peters) Date: Sun, 14 Oct 2001 15:50:36 -0400 Subject: [Python-Dev] conditional expressions (RE: Loop-and-a-half (Re: Curious assignment behaviour)) In-Reply-To: <7xhet39ckb.fsf@ruckus.brouhaha.com> Message-ID: <LNBBLJKPBEHFEDALKOLCIEOCMAAA.tim.one@home.com> [Tim] > If people sign off on taking "then" as a new keyword, I think > the chances are good that we could get > > x = if e1 then e2 else e3 > > into 2.2b1. That's the only obvious spelling, hence the only > truly Pythonic way to spell it. Other languages spelling it that > way range from Algol-60 (Guido's first intense language affair) to > Haskell. [Paul Rubin, among others of similar mind] > This sounds fine to me. Alas, it didn't to Python's parser -- one-token lookahead isn't enough to distinguish if 1: from if 1 then 2 else 3 let alone if a + b / c: from if a + b / c then 2 else 3 etc. and Python won't grow anything a simple parser can't sort out. Everything's cool if parens are required around a conditional expression, though, in which case: x = if e1 then e2 else e3 + 1 # SyntaxError x = (if e1 then e2 else e3) + 1 # cool x = (if e1 then e2 else e3 + 1) # cool x = if e1 then e2 else e3 # SyntaxError x = (if e1 then e2 else e3) # cool x = if if e1 then e2 else e3 then e4 else e5 # SyntaxError x = (if (if e1 then e2 else e3) then e4 else e5) # cool Seems a mixed bag, but I'm more interested in readability and the functionality than in minimizing keystrokes; requiring parens doesn't hurt the goals I care about. implemented-but-not-checked-in-ly y'rs - tim From aahz@panix.com Mon Oct 15 03:27:34 2001 From: aahz@panix.com (aahz@panix.com) Date: Sun, 14 Oct 2001 22:27:34 -0400 (EDT) Subject: [Python-Dev] Will Ware alive? (was Re: where lives uthread (microthreads) now ??) References: <3b091a1c.0110080739.23205b07@posting.google.com> Message-ID: <200110150227.f9F2RYm18907@panix2.panix.com> [x-posted to c.l.py and python-dev] In article <3b091a1c.0110080739.23205b07@posting.google.com>, cmkl <cmkleffner@gmx.de> wrote: > >I wanted to test a samle code with microthreads needed. >The download area: http://world.std.com/~wware/uthread.html >is not available however. > >Is there a mirror to get this code somewhere? Has anybody heard from Will Ware since 9/12? E-mail to him doesn't bounce, but I haven't received any reply. -- --- Aahz <*> (Copyright 2001 by aahz@pobox.com) Hugs and backrubs -- I break Rule 6 http://www.rahul.net/aahz/ Androgynous poly kinky vanilla queer het Pythonista We must not let the evil of a few trample the freedoms of the many. -- --- Aahz <*> (Copyright 2001 by aahz@pobox.com) Hugs and backrubs -- I break Rule 6 http://www.rahul.net/aahz/ Androgynous poly kinky vanilla queer het Pythonista We must not let the evil of a few trample the freedoms of the many. From skip@pobox.com (Skip Montanaro) Mon Oct 15 13:57:46 2001 From: skip@pobox.com (Skip Montanaro) (Skip Montanaro) Date: Mon, 15 Oct 2001 07:57:46 -0500 Subject: [Python-Dev] Problem with Py_NewInterpreter Message-ID: <15306.56650.146816.403758@beluga.mojam.com> Don't know why it got posted where it did (in the mysql-python forum on SF), but some guy posted a question/problem about executing Py_NewInterpreter twice. I have no experience with this sort of stuff, so I figured I'd forward it along to the people with the smarts in this area: http://sourceforge.net/forum/message.php?msg_id=249171 (Need I say, "don't respond to me"? ;-) Skip From jim@interet.com Mon Oct 15 14:40:13 2001 From: jim@interet.com (James C. Ahlstrom) Date: Mon, 15 Oct 2001 09:40:13 -0400 Subject: [Python-Dev] Where is PyMarshal_ReadShortFromFile() ? References: <LNBBLJKPBEHFEDALKOLCKELMMAAA.tim.one@home.com> Message-ID: <3BCAE73D.A1935CAE@interet.com> Tim Peters wrote: > There's a finer distinction to be drawn: "public" doesn't mean "C extern", > ... Ok, thanks for drawing this distinction. So perhaps these functions should have an initial underscore. I have no opinion on this. I assumed the name PyMarshal_ReadLongFromFile() looked very official and was therefore "public". JimA From guido@python.org Mon Oct 15 16:11:45 2001 From: guido@python.org (Guido van Rossum) Date: Mon, 15 Oct 2001 11:11:45 -0400 Subject: [Python-Dev] Problem with Py_NewInterpreter In-Reply-To: Your message of "Mon, 15 Oct 2001 07:57:46 CDT." <15306.56650.146816.403758@beluga.mojam.com> References: <15306.56650.146816.403758@beluga.mojam.com> Message-ID: <200110151511.LAA07344@cj20424-a.reston1.va.home.com> > Don't know why it got posted where it did (in the mysql-python forum on SF), > but some guy posted a question/problem about executing Py_NewInterpreter > twice. I have no experience with this sort of stuff, so I figured I'd > forward it along to the people with the smarts in this area: > > http://sourceforge.net/forum/message.php?msg_id=249171 > > (Need I say, "don't respond to me"? ;-) I've posted a quick pointer to Demo/pysvr which shows how to use Py_NewInterpreter(). It is possible that mysql contains global state that can't be shared between interpreters; mysql may have to be changed to fix that. But I don't know. --Guido van Rossum (home page: http://www.python.org/~guido/) From skip@pobox.com (Skip Montanaro) Mon Oct 15 19:05:19 2001 From: skip@pobox.com (Skip Montanaro) (Skip Montanaro) Date: Mon, 15 Oct 2001 13:05:19 -0500 Subject: [Python-Dev] conditional expressions? Message-ID: <15307.9567.187259.557683@beluga.mojam.com> Is the thread in c.l.py on conditional expressions leading in the direction of actually getting added to the language, or wer Tim's experiments done just to show it's possible? Skip From tim.one@home.com Mon Oct 15 19:25:11 2001 From: tim.one@home.com (Tim Peters) Date: Mon, 15 Oct 2001 14:25:11 -0400 Subject: [Python-Dev] conditional expressions? In-Reply-To: <15307.9567.187259.557683@beluga.mojam.com> Message-ID: <LNBBLJKPBEHFEDALKOLCEEAKMBAA.tim.one@home.com> [Skip Montanaro] > Is the thread in c.l.py on conditional expressions leading in the > direction of actually getting added to the language, or wer Tim's > experiments done just to show it's possible? We (PythonLabs) were wondering whether to reserve any more keywords for 2.2. "then" was a natural candidate, for this specific use. Guido and I have been playing with it. If it proves to be a low-hassle, low-impact change that works well, the intent is to get it in for 2.2b1 later this week. Doesn't look *likely* to me at this point, but don't know. From paul@svensson.org Mon Oct 15 19:30:04 2001 From: paul@svensson.org (Paul Svensson) Date: Mon, 15 Oct 2001 14:30:04 -0400 (EDT) Subject: [Python-Dev] conditional expressions? In-Reply-To: <15307.9567.187259.557683@beluga.mojam.com> Message-ID: <Pine.LNX.4.21.0110151427100.2552-100000@familjen.svensson.org> On Mon, 15 Oct 2001, Skip Montanaro wrote: >Is the thread in c.l.py on conditional expressions leading in the direction >of actually getting added to the language, or wer Tim's experiments done >just to show it's possible? Ick. Please don't tell me that's more than a bad joke ? /Paul From skip@pobox.com (Skip Montanaro) Mon Oct 15 19:46:44 2001 From: skip@pobox.com (Skip Montanaro) (Skip Montanaro) Date: Mon, 15 Oct 2001 13:46:44 -0500 Subject: [Python-Dev] conditional expressions? In-Reply-To: <LNBBLJKPBEHFEDALKOLCEEAKMBAA.tim.one@home.com> References: <15307.9567.187259.557683@beluga.mojam.com> <LNBBLJKPBEHFEDALKOLCEEAKMBAA.tim.one@home.com> Message-ID: <15307.12052.100384.784424@beluga.mojam.com> Tim> [Skip Montanaro] >> Is the thread in c.l.py on conditional expressions leading in the >> direction of actually getting added to the language, or wer Tim's >> experiments done just to show it's possible? Tim> We (PythonLabs) were wondering whether to reserve any more keywords Tim> for 2.2. "then" was a natural candidate, for this specific use. Tim> Guido and I have been playing with it. If it proves to be a Tim> low-hassle, low-impact change that works well, the intent is to get Tim> it in for 2.2b1 later this week. Doesn't look *likely* to me at Tim> this point, but don't know. Regardless whether or not you think this could make it into 2.2b1, I hope if you proceed it will get a PEP a reasonable amount of time before the CVS checkin... <0.1 wink>. It seems downright weird to me that the syntactic baggage necessary to write a conditional expression is greater the the baggage necessary to write an if statement (new "then" keyword, parens required for disambiguation). The parens function pretty much as "{...}" in C, Java, Perl, etc. It's a step away from indentation-based block structure and toward delimiter-based block structure. If you add it, I think it will be harder to justify the lack of delimiter-based block stucture at the statement level. It will just be one more argument in the arsenal of people whose knee-jerk reaction to Python's block structure is to whine about it. Skip From tim.one@home.com Mon Oct 15 19:59:21 2001 From: tim.one@home.com (Tim Peters) Date: Mon, 15 Oct 2001 14:59:21 -0400 Subject: [Python-Dev] conditional expressions? In-Reply-To: <15307.12052.100384.784424@beluga.mojam.com> Message-ID: <LNBBLJKPBEHFEDALKOLCKEANMBAA.tim.one@home.com> [Skip Montanaro] > Regardless whether or not you think this could make it into > 2.2b1, I hope if you proceed it will get a PEP a reasonable > amount of time before the CVS checkin... <0.1 wink>. Guido can exempt himself from the PEP process if he feels like it. If he does, I expect I'll be in the "retroactive PEP" business again. > It seems downright weird to me that the syntactic baggage > necessary to write a conditional expression is greater the the > baggage necessary to write an if statement (new "then" keyword, parens > required for disambiguation). There are many ways a conditional expression could be spelled. This is aiming at the easiest way to read; harder-to-read ways could be introduced with less disruption of the kinds you're focusing on. > The parens function pretty much as "{...}" in C, Java, Perl, etc. Sorry, I don't see the resemblance. > It's a step away from indentation-based block structure and toward > delimiter-based block structure. Ditto. > If you add it, I think it will be harder to justify the lack of > delimiter-based block stucture at the statement level. It will > just be one more argument in the arsenal of people whose knee-jerk > reaction to Python's block structure is to whine about it. I'm not concerned about 21 bad arguments versus 20 <wink>. From skip@pobox.com (Skip Montanaro) Mon Oct 15 20:20:27 2001 From: skip@pobox.com (Skip Montanaro) (Skip Montanaro) Date: Mon, 15 Oct 2001 14:20:27 -0500 Subject: [Python-Dev] conditional expressions? In-Reply-To: <LNBBLJKPBEHFEDALKOLCKEANMBAA.tim.one@home.com> References: <15307.12052.100384.784424@beluga.mojam.com> <LNBBLJKPBEHFEDALKOLCKEANMBAA.tim.one@home.com> Message-ID: <15307.14075.322035.39614@beluga.mojam.com> >> Regardless whether or not you think this could make it into 2.2b1, I >> hope if you proceed it will get a PEP a reasonable amount of time >> before the CVS checkin... <0.1 wink>. Tim> Guido can exempt himself from the PEP process if he feels like it. Tim> If he does, I expect I'll be in the "retroactive PEP" business Tim> again. I'm sure he can, but that doesn't mean he ought to. <wink> >> The parens function pretty much as "{...}" in C, Java, Perl, etc. Tim> Sorry, I don't see the resemblance. Well, the examples I saw looked like x = (if (if e1 then e2 else e3) then e4 else e5) Seems to me that those inner parens are there to separate the inner conditional from the trailing parts of the outer conditional: x = ( if ( if ( e1 then e2 else e3 ) then e4 else e5 ) Whether or not that's what they are in a language parser sense, they sure look like it to the human eye. >> If you add it, I think it will be harder to justify the lack of >> delimiter-based block stucture at the statement level. It will just >> be one more argument in the arsenal of people whose knee-jerk >> reaction to Python's block structure is to whine about it. Tim> I'm not concerned about 21 bad arguments versus 20 <wink>. That 21st argument will be a little stronger than the other, because it will go something like, "Why can't we have delimiters for block statements? After all, we have them in conditional expressions." The reference, instead of being to another language, will be to Python itself. Skip From guido@python.org Mon Oct 15 20:24:03 2001 From: guido@python.org (Guido van Rossum) Date: Mon, 15 Oct 2001 15:24:03 -0400 Subject: [Python-Dev] conditional expressions? In-Reply-To: Your message of "Mon, 15 Oct 2001 13:46:44 CDT." <15307.12052.100384.784424@beluga.mojam.com> References: <15307.9567.187259.557683@beluga.mojam.com> <LNBBLJKPBEHFEDALKOLCEEAKMBAA.tim.one@home.com> <15307.12052.100384.784424@beluga.mojam.com> Message-ID: <200110151924.PAA25209@cj20424-a.reston1.va.home.com> [Skip] > >> Is the thread in c.l.py on conditional expressions leading in the > >> direction of actually getting added to the language, or wer Tim's > >> experiments done just to show it's possible? [Paul Svensson] > Ick. Please don't tell me that's more than a bad joke ? [Tim] > Tim> We (PythonLabs) were wondering whether to reserve any more > Tim> keywords for 2.2. "then" was a natural candidate, for this > Tim> specific use. Guido and I have been playing with it. If > Tim> it proves to be a low-hassle, low-impact change that works > Tim> well, the intent is to get it in for 2.2b1 later this week. > Tim> Doesn't look *likely* to me at this point, but don't know. [Skip] > Regardless whether or not you think this could make it into 2.2b1, I > hope if you proceed it will get a PEP a reasonable amount of time > before the CVS checkin... <0.1 wink>. > > It seems downright weird to me that the syntactic baggage necessary > to write a conditional expression is greater the the baggage > necessary to write an if statement (new "then" keyword, parens > required for disambiguation). The parens function pretty much as > "{...}" in C, Java, Perl, etc. It's a step away from > indentation-based block structure and toward delimiter-based block > structure. If you add it, I think it will be harder to justify the > lack of delimiter-based block stucture at the statement level. It > will just be one more argument in the arsenal of people whose > knee-jerk reaction to Python's block structure is to whine about it. I think you must be misunderstanding the proposal, which is to add if <expr> then <expr> else <expr> as an alternative to the expression syntax. Here's a preliminary patch (which I won't apply until I have more confidence that this is acceptable to the community): http://sourceforge.net/tracker/index.php?func=detail&aid=471421&group_id=5470&atid=305470 The parens in this proposal (my version) act no different than any other use of parentheses in Python expressions. Basically, you need to add parentheses to disambiguate expressions: - if otherwise the 'if' keyword would be the start of a statement (because 'if' at the start of a statement starts an if *statement*, and the parser can't look ahead for the 'then' keyword); - if the conditional expression is to be combined with a unary or binary operator. Some examples where no parentheses are needed (note that a comma binds less tight than a conditional expression -- same as for lambda): x = if 1 then 2 else 3, y f(if 1 then 2 else 3, y) a[if 1 then 2 else 3, y] `if 1 then 2 else 3` lambda: if 1 then 2 else 3 Some examples where parentheses *are* required: (if 1 then 2 else 3) + 4 a[(if i then 2 else 3) : 4] In some situations I'm not sure what's right; The un-parenthesized form looks weird although it's not neede to avoid ambiguity: if (if 1 then 2 else 3): pass print (if 1 then 2 else 3) for i in (if 1 then "abc" else "def"): pass I'd be happy to discuss this more. I'm *not* happy with responses like "is this a bad joke?". I don't understand how this could be an argument in the arsenal of the anti-Python league. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Mon Oct 15 20:34:19 2001 From: guido@python.org (Guido van Rossum) Date: Mon, 15 Oct 2001 15:34:19 -0400 Subject: [Python-Dev] conditional expressions? In-Reply-To: Your message of "Mon, 15 Oct 2001 14:20:27 CDT." <15307.14075.322035.39614@beluga.mojam.com> References: <15307.12052.100384.784424@beluga.mojam.com> <LNBBLJKPBEHFEDALKOLCKEANMBAA.tim.one@home.com> <15307.14075.322035.39614@beluga.mojam.com> Message-ID: <200110151934.PAA25299@cj20424-a.reston1.va.home.com> > Well, the examples I saw looked like > > x = (if (if e1 then e2 else e3) then e4 else e5) > > Seems to me that those inner parens are there to separate the inner > conditional from the trailing parts of the outer conditional: > > x = ( > if ( > if ( ^ this is not there > e1 > then > e2 > else > e3 > ) > then > e4 > else > e5 > ) > > Whether or not that's what they are in a language parser sense, they sure > look like it to the human eye. They are syntactically unnecessary; they are mostly for guidance of the human reader. If I saw a piece of code that read x = if if if x == 1 then y else z then p else q then a else b I would get a strong urge to commit illegal violence against the author. If on the other hand I saw x = if (if (if x == 1 then y else z) then p else q) then a else b I might be willing to sit down and figure out what it meant -- maybe with the help of a parentheses-balancing command in my editor. > Tim> I'm not concerned about 21 bad arguments versus 20 <wink>. > > That 21st argument will be a little stronger than the other, because > it will go something like, "Why can't we have delimiters for block > statements? After all, we have them in conditional expressions." > The reference, instead of being to another language, will be to > Python itself. Sorry, I still don't get this at all. A conditional expression is still an expression. Curly brances are a statement-level concept. What am I missing? Where is the similarity between the use of { } and ( ) in C/Java/C++/Perl? --Guido van Rossum (home page: http://www.python.org/~guido/) From paul-python@svensson.org Mon Oct 15 21:04:59 2001 From: paul-python@svensson.org (Paul Svensson) Date: Mon, 15 Oct 2001 16:04:59 -0400 (EDT) Subject: [Python-Dev] conditional expressions? In-Reply-To: <200110151924.PAA25209@cj20424-a.reston1.va.home.com> Message-ID: <Pine.LNX.4.21.0110151552470.2552-100000@familjen.svensson.org> On Mon, 15 Oct 2001, Guido van Rossum wrote: >I'd be happy to discuss this more. I'm *not* happy with responses >like "is this a bad joke?". I don't understand how this could be an >argument in the arsenal of the anti-Python league. I'm sorry if it hit a bad nerve, but I did skip this whole thread on c.l.py, thinking that this could surely not be seriously meant, so I guess don't have the understanding of the motivation behind the proposal that I could have had. I know there's been a long time request for this feature, supposedly from people coming to Python from C, but what's the real reason for adding it now, without even a PEP ? Does it really add enough to the language to justify the bloat ? /Paul From skip@pobox.com (Skip Montanaro) Mon Oct 15 21:13:48 2001 From: skip@pobox.com (Skip Montanaro) (Skip Montanaro) Date: Mon, 15 Oct 2001 15:13:48 -0500 Subject: [Python-Dev] conditional expressions? In-Reply-To: <200110151934.PAA25299@cj20424-a.reston1.va.home.com> References: <15307.12052.100384.784424@beluga.mojam.com> <LNBBLJKPBEHFEDALKOLCKEANMBAA.tim.one@home.com> <15307.14075.322035.39614@beluga.mojam.com> <200110151934.PAA25299@cj20424-a.reston1.va.home.com> Message-ID: <15307.17276.738359.631642@beluga.mojam.com> I misunderstood about the need for parens. I had only been watching the thread in python-list with one eye. I thought it was just some python-list riff-raff throwing around ideas. Then I saw that Tim was part of the rabble ;-). I still didn't notice the "I've implemented it but not yet checked it in" bit until someone else did and asked about it. I then went looking for a PEP but didn't see one. I hadn't seen any discussion on python-dev either. (Maybe I missed it because python-list and python-dev mail wind up in the same mailbox.) Guido> If I saw a piece of code that read Guido> x = if if if x == 1 then y else z then p else q then a else b Guido> I would get a strong urge to commit illegal violence against the Guido> author. If on the other hand I saw Guido> x = if (if (if x == 1 then y else z) then p else q) then a else b Guido> I might be willing to sit down and figure out what it meant -- Guido> maybe with the help of a parentheses-balancing command in my Guido> editor. Python's block structured statement syntax keep you from composing just monstrosities at the statement level. I'm not sure what makes them more appealing at the expression level. How many levels of parens would be required before you had a "strong urge to commit illegal violence" against someone using a parenthesized conditional expression? I guess that's what it boils down to for me. You can easily write much more inscrutable code with the new conditional expression than you could before. I think that will make Python somewhat less of a CP4E language. I don't think it falls into the same category of the more esoteric features of the language (generators, metaclasses, __*__ methods, etc) that beginners can safely ignore. Conditional expressions are pretty basic stuff, and this makes it easier to write fairly inscrutable conditional expressions. Guido> Sorry, I still don't get this at all. A conditional expression Guido> is still an expression. Curly brances are a statement-level Guido> concept. What am I missing? Where is the similarity between the Guido> use of { } and ( ) in C/Java/C++/Perl? >From the computer's viewpoint, they are syntactically and semantically, different, I agree. From the point of view of a human trying to understand what's written I think they will serve much the same purpose, however. White space is viewed is being essential enough to understanding that Python uses it for block structure. In particular, I can't nest two compound statements on the same line. With this new addition to expressions, it appears you will be able to create arbitrarily complex expressions all on one line. Skip From guido@python.org Mon Oct 15 21:14:48 2001 From: guido@python.org (Guido van Rossum) Date: Mon, 15 Oct 2001 16:14:48 -0400 Subject: [Python-Dev] conditional expressions? In-Reply-To: Your message of "Mon, 15 Oct 2001 16:04:59 EDT." <Pine.LNX.4.21.0110151552470.2552-100000@familjen.svensson.org> References: <Pine.LNX.4.21.0110151552470.2552-100000@familjen.svensson.org> Message-ID: <200110152014.QAA25765@cj20424-a.reston1.va.home.com> [Paul] > I know there's been a long time request for this feature, > supposedly from people coming to Python from C, or any other language -- it's in Java, Perl, C++, Icon, Haskell, Lisp... People have been arguing for years how to do this. > but what's the real reason for adding it now, > without even a PEP ? > Does it really add enough to the language to justify the bloat ? We'll see about the PEP (I think it's a good idea to motivate the change). The language bloat is very small IMO -- much less than augmented assignment or list comprehensions, actually, and the benefits are quite substantial, when used judiciously. Like augmented assignment, the conditional expression matches how I *think* about certain situations; it's nice to finally be able to write them down right. It's especially useful in function arguments, in string formatting operations, or in lambdas (if you like lambdas :-). --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Mon Oct 15 21:18:48 2001 From: guido@python.org (Guido van Rossum) Date: Mon, 15 Oct 2001 16:18:48 -0400 Subject: [Python-Dev] conditional expressions? In-Reply-To: Your message of "Mon, 15 Oct 2001 15:13:48 CDT." <15307.17276.738359.631642@beluga.mojam.com> References: <15307.12052.100384.784424@beluga.mojam.com> <LNBBLJKPBEHFEDALKOLCKEANMBAA.tim.one@home.com> <15307.14075.322035.39614@beluga.mojam.com> <200110151934.PAA25299@cj20424-a.reston1.va.home.com> <15307.17276.738359.631642@beluga.mojam.com> Message-ID: <200110152018.QAA25787@cj20424-a.reston1.va.home.com> > I misunderstood about the need for parens. I had only been watching the > thread in python-list with one eye. I thought it was just some python-list > riff-raff throwing around ideas. Then I saw that Tim was part of the rabble > ;-). I still didn't notice the "I've implemented it but not yet checked it > in" bit until someone else did and asked about it. I then went looking for > a PEP but didn't see one. I hadn't seen any discussion on python-dev > either. (Maybe I missed it because python-list and python-dev mail wind up > in the same mailbox.) OK. I guess you were thinking that the parentheses were required as for 'if' in C? > Guido> If I saw a piece of code that read > > Guido> x = if if if x == 1 then y else z then p else q then a else b > > Guido> I would get a strong urge to commit illegal violence against the > Guido> author. If on the other hand I saw > > Guido> x = if (if (if x == 1 then y else z) then p else q) then a else b > > Guido> I might be willing to sit down and figure out what it meant -- > Guido> maybe with the help of a parentheses-balancing command in my > Guido> editor. > > Python's block structured statement syntax keep you from composing just > monstrosities at the statement level. I'm not sure what makes them more > appealing at the expression level. How many levels of parens would be > required before you had a "strong urge to commit illegal violence" against > someone using a parenthesized conditional expression? As soon as there is an if inside the condition of the first if. Even parentheses don't fix this, really. > I guess that's what it boils down to for me. You can easily write > much more inscrutable code with the new conditional expression than > you could before. "You can write Fortran in any language" -- we've been through this before. Potential abuse is not enough to prevent it -- likely abuse is. > I think that will make Python somewhat less of a CP4E language. I > don't think it falls into the same category of the more esoteric > features of the language (generators, metaclasses, __*__ methods, > etc) that beginners can safely ignore. Conditional expressions are > pretty basic stuff, and this makes it easier to write fairly > inscrutable conditional expressions. It also makes it easier to write what you thinkk, IMO. > Guido> Sorry, I still don't get this at all. A conditional > Guido> expression is still an expression. Curly brances are a > Guido> statement-level concept. What am I missing? Where is > Guido> the similarity between the use of { } and ( ) in > Guido> C/Java/C++/Perl? > > From the computer's viewpoint, they are syntactically and > semantically, different, I agree. From the point of view of a human > trying to understand what's written I think they will serve much the > same purpose, however. White space is viewed is being essential > enough to understanding that Python uses it for block structure. In > particular, I can't nest two compound statements on the same line. > With this new addition to expressions, it appears you will be able > to create arbitrarily complex expressions all on one line. Previous versions of Python in no way impaired your ability to create arbitrarily complex expressions all on one line. --Guido van Rossum (home page: http://www.python.org/~guido/) From tim.one@home.com Mon Oct 15 21:24:05 2001 From: tim.one@home.com (Tim Peters) Date: Mon, 15 Oct 2001 16:24:05 -0400 Subject: [Python-Dev] conditional expressions? In-Reply-To: <200110151924.PAA25209@cj20424-a.reston1.va.home.com> Message-ID: <LNBBLJKPBEHFEDALKOLCOEBGMBAA.tim.one@home.com> [Guido, to Skip] > I think you must be misunderstanding the proposal Skip was responding to the c.l.py discussion, which you probably haven't seen. I started it there sometime over the weekend, and "the rules" at that time required parens in all cases (which is just what was implemented at the time). From paul-python@svensson.org Mon Oct 15 21:39:30 2001 From: paul-python@svensson.org (Paul Svensson) Date: Mon, 15 Oct 2001 16:39:30 -0400 (EDT) Subject: [Python-Dev] conditional expressions? In-Reply-To: <200110151934.PAA25299@cj20424-a.reston1.va.home.com> Message-ID: <Pine.LNX.4.21.0110151608190.2552-100000@familjen.svensson.org> On Mon, 15 Oct 2001, Guido van Rossum wrote: >Sorry, I still don't get this at all. A conditional expression is >still an expression. Curly brances are a statement-level concept. >What am I missing? Where is the similarity between the use of { } and >( ) in C/Java/C++/Perl? Trying to put a finger on why I think this proposed new syntax looks ugly, I would say that my biggest problem is that it blurs some of the visual distictions between statements and expressions that I find helpful for quickly locating the interesting parts of a piece of code. I assume this has some parallel to list comprehensions that way, but I fail to see the same blatant conceptual beauty in this proposal, motivating the unfamiliar form. /Paul From guido@python.org Mon Oct 15 21:42:57 2001 From: guido@python.org (Guido van Rossum) Date: Mon, 15 Oct 2001 16:42:57 -0400 Subject: [Python-Dev] conditional expressions? In-Reply-To: Your message of "Mon, 15 Oct 2001 16:39:30 EDT." <Pine.LNX.4.21.0110151608190.2552-100000@familjen.svensson.org> References: <Pine.LNX.4.21.0110151608190.2552-100000@familjen.svensson.org> Message-ID: <200110152042.QAA28987@cj20424-a.reston1.va.home.com> > Trying to put a finger on why I think this proposed new syntax looks ugly, > I would say that my biggest problem is that it blurs some of the visual > distictions between statements and expressions that I find helpful > for quickly locating the interesting parts of a piece of code. > > I assume this has some parallel to list comprehensions that way, > but I fail to see the same blatant conceptual beauty in this proposal, > motivating the unfamiliar form. And what exactly qualifies you as a language designer? --Guido van Rossum (home page: http://www.python.org/~guido/) From skip@pobox.com (Skip Montanaro) Mon Oct 15 21:50:40 2001 From: skip@pobox.com (Skip Montanaro) (Skip Montanaro) Date: Mon, 15 Oct 2001 15:50:40 -0500 Subject: [Python-Dev] New socketmodule.c warnings Message-ID: <15307.19488.702413.550932@beluga.mojam.com> Just came across these new warnings (gcc 2.96, Linux Mandrake): .../Modules/socketmodule.c:1625: warning: initialization from incompatible pointer type .../Modules/socketmodule.c:1629: warning: initialization from incompatible pointer type .../Modules/socketmodule.c:1636: warning: initialization from incompatible pointer type .../Modules/socketmodule.c:1639: warning: initialization from incompatible pointer type .../Modules/socketmodule.c:1642: warning: initialization from incompatible pointer type .../Modules/socketmodule.c:1645: warning: initialization from incompatible pointer type I see a lot of interface changes in socketmodule between 1.175 and 1.176. Is this a spurious error or something significant? Skip From skip@pobox.com (Skip Montanaro) Mon Oct 15 21:57:47 2001 From: skip@pobox.com (Skip Montanaro) (Skip Montanaro) Date: Mon, 15 Oct 2001 15:57:47 -0500 Subject: [Python-Dev] New socketmodule.c warnings In-Reply-To: <15307.19488.702413.550932@beluga.mojam.com> References: <15307.19488.702413.550932@beluga.mojam.com> Message-ID: <15307.19915.436629.47120@beluga.mojam.com> Skip> Is this a spurious error or something significant? Oops... s/error/warning/ S From gmcm@hypernet.com Mon Oct 15 22:24:29 2001 From: gmcm@hypernet.com (Gordon McMillan) Date: Mon, 15 Oct 2001 17:24:29 -0400 Subject: [Python-Dev] conditional expressions? In-Reply-To: <Pine.LNX.4.21.0110151552470.2552-100000@familjen.svensson.org> References: <200110151924.PAA25209@cj20424-a.reston1.va.home.com> Message-ID: <3BCB1BCD.7229.9CA08A91@localhost> Paul Svensson wrote: > so I guess don't have the understanding of the motivation > behind the proposal that I could have had. Because: a and b or c burns you when b evaluates to false, but (a and [b] or [c])[0] is a monstrosity. And yes, people do that. - Gordon From tim.one@home.com Mon Oct 15 22:25:43 2001 From: tim.one@home.com (Tim Peters) Date: Mon, 15 Oct 2001 17:25:43 -0400 Subject: [Python-Dev] conditional expressions? In-Reply-To: <200110151924.PAA25209@cj20424-a.reston1.va.home.com> Message-ID: <LNBBLJKPBEHFEDALKOLCKEBLMBAA.tim.one@home.com> [Guido] > ... > Some examples where no parentheses are needed (note that a comma binds > less tight than a conditional expression -- same as for lambda): > > x = if 1 then 2 else 3, y Also x = if 1 then 2 else 3, which sets x to the singleton tuple (2,). This is an old potential confusion, though, much the same as the current x = 2 or 3, (which also sets x to (2,)). > ... > Some examples where parentheses *are* required: > > (if 1 then 2 else 3) + 4 But that's because it's a top-level expression beginning with "if", not because of the "+", right? In the patched Python, this works fine: >>> x = if 0 then 2 else 3 + 4 >>> x 7 >>> OTOH, these seem odd: >>> 4 + if 1 then 2 else 3 File "<stdin>", line 1 4 + if 1 then 2 else 3 ^ SyntaxError: invalid syntax >>> x = 4 + if 1 then 2 else 3 File "<stdin>", line 1 x = 4 + if 1 then 2 else 3 ^ SyntaxError: invalid syntax >>> I say "odd" because there's no question about what they mean. > a[(if i then 2 else 3) : 4] > > In some situations I'm not sure what's right; The un-parenthesized > form looks weird although it's not neede to avoid ambiguity: > > if (if 1 then 2 else 3): pass > print (if 1 then 2 else 3) OTOH, print >> sys.stderr, if 1 then 2 else 3 and print 4, if 1 then 2 else 3 don't look particularly weird but also raise SyntaxError. As a matter of taste, I was most surprised by the 4 + if 1 then 2 else 3 examples. From fredrik@pythonware.com Mon Oct 15 22:47:54 2001 From: fredrik@pythonware.com (Fredrik Lundh) Date: Mon, 15 Oct 2001 23:47:54 +0200 Subject: [Python-Dev] conditional expressions? References: <15307.12052.100384.784424@beluga.mojam.com> <LNBBLJKPBEHFEDALKOLCKEANMBAA.tim.one@home.com> <15307.14075.322035.39614@beluga.mojam.com> <200110151934.PAA25299@cj20424-a.reston1.va.home.com> <15307.17276.738359.631642@beluga.mojam.com> Message-ID: <00b001c155c3$0ca44380$ced241d5@hagrid> skip wrote: > I think that will make Python somewhat less of a CP4E language. I doubt Python 2.2 will qualify as a CP4E language, even if conditional expressions are left out. </F> From guido@python.org Mon Oct 15 22:42:39 2001 From: guido@python.org (Guido van Rossum) Date: Mon, 15 Oct 2001 17:42:39 -0400 Subject: [Python-Dev] conditional expressions? In-Reply-To: Your message of "Mon, 15 Oct 2001 17:25:43 EDT." <LNBBLJKPBEHFEDALKOLCKEBLMBAA.tim.one@home.com> References: <LNBBLJKPBEHFEDALKOLCKEBLMBAA.tim.one@home.com> Message-ID: <200110152142.RAA02911@cj20424-a.reston1.va.home.com> > [Guido] > > ... > > Some examples where no parentheses are needed (note that a comma binds > > less tight than a conditional expression -- same as for lambda): > > > > x = if 1 then 2 else 3, y [Tim] > Also > > x = if 1 then 2 else 3, > > which sets x to the singleton tuple (2,). This is an old potential > confusion, though, much the same as the current > > x = 2 or 3, > > (which also sets x to (2,)). These don't bother me any more than x = 2, does. > > ... > > Some examples where parentheses *are* required: > > > > (if 1 then 2 else 3) + 4 > > But that's because it's a top-level expression beginning with "if", not > because of the "+", right? In the patched Python, this works fine: > > >>> x = if 0 then 2 else 3 + 4 > >>> x > 7 > >>> But it groups differently: the 'else' part is '3 + 4' here. Try 'if 1'. :-) > OTOH, these seem odd: > > >>> 4 + if 1 then 2 else 3 > File "<stdin>", line 1 > 4 + if 1 then 2 else 3 > ^ > SyntaxError: invalid syntax That's too much work for the parser to get right. And you'd get anomalies like 3 * if 1 then 2 else 3 + 4 -- how should this group? > >>> x = 4 + if 1 then 2 else 3 > File "<stdin>", line 1 > x = 4 + if 1 then 2 else 3 > ^ > SyntaxError: invalid syntax > >>> > > I say "odd" because there's no question about what they mean. Only because the examples are so simply. > > a[(if i then 2 else 3) : 4] > > > > In some situations I'm not sure what's right; The un-parenthesized > > form looks weird although it's not neede to avoid ambiguity: > > > > if (if 1 then 2 else 3): pass > > print (if 1 then 2 else 3) > > OTOH, > > print >> sys.stderr, if 1 then 2 else 3 > and > print 4, if 1 then 2 else 3 > > don't look particularly weird but also raise SyntaxError. > > As a matter of taste, I was most surprised by the > > 4 + if 1 then 2 else 3 > > examples. They act the same as lambda in this respect. Because the conditional *ends* in something that "pulls in" all following binary operators and their operands until it hits a ")", I find it safer not to let it accept an operator at the front. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Mon Oct 15 23:02:31 2001 From: guido@python.org (Guido van Rossum) Date: Mon, 15 Oct 2001 18:02:31 -0400 Subject: [Python-Dev] conditional expressions? In-Reply-To: Your message of "Mon, 15 Oct 2001 17:42:39 EDT." <200110152142.RAA02911@cj20424-a.reston1.va.home.com> References: <LNBBLJKPBEHFEDALKOLCKEBLMBAA.tim.one@home.com> <200110152142.RAA02911@cj20424-a.reston1.va.home.com> Message-ID: <200110152202.SAA04326@cj20424-a.reston1.va.home.com> > Because the conditional *ends* in something that "pulls in" all > following binary operators and their operands until it hits a ")", > I find it safer not to let it accept an operator at the front. To explain this perhaps a bit better, the 'else' keyword acts as the right side of a low-priority binary operator with respect to the expression that follows. It would be unpleasant if the 'if' keyword acted as the left side of a *high*-priority operator to the expression that preceeds it. Again, think of what this would mean: (3 * if x then y else z + 4) --Guido van Rossum (home page: http://www.python.org/~guido/) From pedroni@inf.ethz.ch Mon Oct 15 23:29:54 2001 From: pedroni@inf.ethz.ch (Samuele Pedroni) Date: Tue, 16 Oct 2001 00:29:54 +0200 Subject: [Python-Dev] conditional expressions? References: <15307.12052.100384.784424@beluga.mojam.com> <LNBBLJKPBEHFEDALKOLCKEANMBAA.tim.one@home.com> <15307.14075.322035.39614@beluga.mojam.com> <200110151934.PAA25299@cj20424-a.reston1.va.home.com> <15307.17276.738359.631642@beluga.mojam.com> <00b001c155c3$0ca44380$ced241d5@hagrid> Message-ID: <003601c155c8$eae0b660$8e0bfea9@newmexico> [Fredrik Lundh] > > I doubt Python 2.2 will qualify as a CP4E language, even if > conditional expressions are left out. > [sorry but I feel I should say this] I heartly agree, i.e. I'm fine with that but it is true. regards. From tim.one@home.com Tue Oct 16 00:30:58 2001 From: tim.one@home.com (Tim Peters) Date: Mon, 15 Oct 2001 19:30:58 -0400 Subject: [Python-Dev] conditional expressions? In-Reply-To: <200110152142.RAA02911@cj20424-a.reston1.va.home.com> Message-ID: <LNBBLJKPBEHFEDALKOLCEECFMBAA.tim.one@home.com> ||| ... ||| Some examples where parentheses *are* required: ||| ||| (if 1 then 2 else 3) + 4 || But that's because it's a top-level expression beginning with "if", not || because of the "+", right? In the patched Python, this works fine: || || >>> x = if 0 then 2 else 3 + 4 || >>> x || 7 || >>> | But it groups differently: the 'else' part is '3 + 4' here. Try 'if | 1'. :-) Yes, I understand how it groups, and am fine with it. The question I asked is the one I asked <wink>: the paretheses aren't required because of the "+", but solely because it's a "top-level expression". This is by way of contrast with the example that was next to it in the original: a[(if i then 2 else 3) : 4] where, unlike the "+" example, the parens are also required if you stick that on the RHS of an assignment. I thought both were *intended* to be examples of: you need to add parentheses to disambiguate expressions: ... - if the conditional expression is to be combined with a unary or binary operator. as "+" and ":" both feel like <wink> binary operators in these examples. || OTOH, these seem odd: || || >>> 4 + if 1 then 2 else 3 || File "<stdin>", line 1 || 4 + if 1 then 2 else 3 || ^ || SyntaxError: invalid syntax > That's too much work for the parser to get right. And you'd get > anomalies like > > 3 * if 1 then 2 else 3 + 4 > > -- how should this group? That's a good point. > ... > Because the conditional *ends* in something that "pulls in" all > following binary operators and their operands until it hits a ")", > I find it safer not to let it accept an operator at the front. I agree. The "precedence pull" of "*" does make it more tempting to misread x = 3 * if 1 then 2 else 3 + 4 than to misread x = if 1 then 2 else 3 + 4 and I'm personally happier with "excess" parentheses anyway. From tim.one@home.com Tue Oct 16 04:22:51 2001 From: tim.one@home.com (Tim Peters) Date: Mon, 15 Oct 2001 23:22:51 -0400 Subject: [Python-Dev] conditional expressions? In-Reply-To: <Pine.LNX.4.21.0110151552470.2552-100000@familjen.svensson.org> Message-ID: <LNBBLJKPBEHFEDALKOLCAECMMBAA.tim.one@home.com> [Paul Svensson] > ... > I know there's been a long time request for this feature, > supposedly from people coming to Python from C, > but what's the real reason for adding it now, > without even a PEP ? The reason for *considering* it right now is a rapidly closing window of opportunity -- 2.2b1 goes out later this week. A fellow on c.l.py has been plugging away at a PEP, Eric Raymond submitted a working patch for C's ?: in January of 2000, Jim Fulton loved the idea then (but not particularly the ?: spelling), David Ascher reported that lack of a conditional was a FAQ in his classes then, etc etc. Most replies hated the ?: syntax, and Guido eventually remembered (although I think he forgot it this time around) that Algol-60 used the obvious if/then/else. His reply on Mon, 31 Jan 2000 09:36:52 -0500: The solution can be the same as what Algol used: 'if' outside parentheses is a statement, and inside parentheses is an expression. It's a bit of a grammar rearrangement, but totally unambiguous. However, the added keyword means it won't be in 1.6. The lively discussion means that Eric's patch will have a hard time getting in too... The only difference between then and now is s/1.6/2.2/. > Does it really add enough to the language to justify the bloat ? I expect most people would say it does; by definition I'll adjust my beliefs to whatever Guido decides <wink>. From mahler@cyc.com Tue Oct 16 04:40:47 2001 From: mahler@cyc.com (Daniel Mahler) Date: Mon, 15 Oct 2001 22:40:47 -0500 Subject: [Python-Dev] conditional expressions? In-Reply-To: <LNBBLJKPBEHFEDALKOLCAECMMBAA.tim.one@home.com> References: <Pine.LNX.4.21.0110151552470.2552-100000@familjen.svensson.org> <LNBBLJKPBEHFEDALKOLCAECMMBAA.tim.one@home.com> Message-ID: <15307.44095.944381.833422@mcallister.cyc.com> Has an if(cond,exp1,exp2) syntax been considered? It would have the advantage that existing syntax sensitive editors would handle without change. I think conditional exprs badly needed, but the if/then/else syntax seems too verbose for use inside expressions. regards Daniel Mahler Tim Peters writes: > [Paul Svensson] > > ... > > I know there's been a long time request for this feature, > > supposedly from people coming to Python from C, > > but what's the real reason for adding it now, > > without even a PEP ? > > The reason for *considering* it right now is a rapidly closing window of > opportunity -- 2.2b1 goes out later this week. A fellow on c.l.py has been > plugging away at a PEP, Eric Raymond submitted a working patch for C's ?: in > January of 2000, Jim Fulton loved the idea then (but not particularly the ?: > spelling), David Ascher reported that lack of a conditional was a FAQ in his > classes then, etc etc. Most replies hated the ?: syntax, and Guido > eventually remembered (although I think he forgot it this time around) that > Algol-60 used the obvious if/then/else. His reply on Mon, 31 Jan 2000 > 09:36:52 -0500: > > The solution can be the same as what Algol used: 'if' outside > parentheses is a statement, and inside parentheses is an expression. > It's a bit of a grammar rearrangement, but totally unambiguous. > > However, the added keyword means it won't be in 1.6. The lively > discussion means that Eric's patch will have a hard time getting > in too... > > The only difference between then and now is s/1.6/2.2/. > > > Does it really add enough to the language to justify the bloat ? > > I expect most people would say it does; by definition I'll adjust my beliefs > to whatever Guido decides <wink>. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev From tim.one@home.com Tue Oct 16 05:41:50 2001 From: tim.one@home.com (Tim Peters) Date: Tue, 16 Oct 2001 00:41:50 -0400 Subject: [Python-Dev] conditional expressions? In-Reply-To: <15307.44095.944381.833422@mcallister.cyc.com> Message-ID: <LNBBLJKPBEHFEDALKOLCIECPMBAA.tim.one@home.com> [Daniel Mahler] > Has an if(cond,exp1,exp2) syntax been considered? Not seriously, no. That looks like a function call, and function arguments are always evaluated in Python, and that works against seeing this as a short-circuiting construct: q = if y then x/y else copysign(INFINITY, x) > It would have the advantage that existing > syntax sensitive editors would handle without change. I don't see any problem there (and am responsible for two context sensitive Python edit gimmicks, and used to be responsible for a third -- all I expect to need to do is get "then" at token level colored as keyword). > I think conditional exprs badly needed, > but the if/then/else syntax seems too verbose for use inside > expressions. Good -- then you won't be tempted to abuse it <0.5 wink>. best-in-small-doses-ly y'rs - tim From greg@cosc.canterbury.ac.nz Tue Oct 16 06:06:40 2001 From: greg@cosc.canterbury.ac.nz (Greg Ewing) Date: Tue, 16 Oct 2001 18:06:40 +1300 (NZDT) Subject: [Python-Dev] conditional expressions? In-Reply-To: <15307.12052.100384.784424@beluga.mojam.com> Message-ID: <200110160506.SAA07491@s454.cosc.canterbury.ac.nz> Skip Montanaro <skip@pobox.com>: > It seems downright weird to me that the syntactic baggage necessary to write > a conditional expression is greater the the baggage necessary to write an if > statement (new "then" keyword, parens required for disambiguation). Has the "x if c else y" idea been totally rejected? It doesn't require a new keyword, or parentheses to disambiguate from an if statement. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From tim.one@home.com Tue Oct 16 06:15:05 2001 From: tim.one@home.com (Tim Peters) Date: Tue, 16 Oct 2001 01:15:05 -0400 Subject: [Python-Dev] conditional expressions? In-Reply-To: <200110160506.SAA07491@s454.cosc.canterbury.ac.nz> Message-ID: <LNBBLJKPBEHFEDALKOLCMEDAMBAA.tim.one@home.com> [Greg Ewing] > Has the "x if c else y" idea been totally rejected? > It doesn't require a new keyword, or parentheses to > disambiguate from an if statement. I was its only fan within PythonLabs, and I was lukewarm. The trick is to view "if c else" as a new short-circuiting infix binary operator (for each c), and then it makes great-- even elegant --sense. But it's apparently a major strain to see it that way. It had at least one sworn mortal enemy, and I think Fred Drake was so upset at the idea that he couldn't speak. bad-sign-in-a-wordsmith<wink>-ly y'rs - tim From esr@thyrsus.com Tue Oct 16 06:54:32 2001 From: esr@thyrsus.com (Eric S. Raymond) Date: Tue, 16 Oct 2001 01:54:32 -0400 Subject: [Python-Dev] conditional expressions? In-Reply-To: <Pine.LNX.4.21.0110151552470.2552-100000@familjen.svensson.org>; from paul-python@svensson.org on Mon, Oct 15, 2001 at 04:04:59PM -0400 References: <200110151924.PAA25209@cj20424-a.reston1.va.home.com> <Pine.LNX.4.21.0110151552470.2552-100000@familjen.svensson.org> Message-ID: <20011016015432.E3779@thyrsus.com> Paul Svensson <paul-python@svensson.org>: > On Mon, 15 Oct 2001, Guido van Rossum wrote: > > >I'd be happy to discuss this more. I'm *not* happy with responses > >like "is this a bad joke?". I don't understand how this could be an > >argument in the arsenal of the anti-Python league. > > I'm sorry if it hit a bad nerve, > but I did skip this whole thread on c.l.py, > thinking that this could surely not be seriously meant, > so I guess don't have the understanding of the motivation > behind the proposal that I could have had. > > I know there's been a long time request for this feature, > supposedly from people coming to Python from C, > but what's the real reason for adding it now, > without even a PEP ? > Does it really add enough to the language to justify the bloat ? I actually implemented this last year. The less cumbersome syntax I suggest is: a ? b ! c -- <a href="http://www.tuxedo.org/~esr/">Eric S. Raymond</a> Everything you know is wrong. But some of it is a useful first approximation. From pf@artcom-gmbh.de Tue Oct 16 08:06:41 2001 From: pf@artcom-gmbh.de (Peter Funk) Date: Tue, 16 Oct 2001 09:06:41 +0200 (CEST) Subject: [Python-Dev] conditional expressions? In-Reply-To: <LNBBLJKPBEHFEDALKOLCAECMMBAA.tim.one@home.com> from Tim Peters at "Oct 15, 2001 11:22:51 pm" Message-ID: <m15tOJF-002XjiC@artcom0.artcom-gmbh.de> Hi, I'm -1 on the concept of conditional expressions. And I don't give a damn whether they are spelled x?y:z or (if x then y else z). Let me explain why: IMO the motivation of people asking for conditional expressions is, that they want to write down complex constructs without having to name these properly. Otherwise it is easy to put the code into a function or method and call this instead. In most cases this will improve the readability of the code considerable, since people are alreaday trained to choose proper identifiers instead of X, Y, Z. Conditional expressions will be abused as an obfuscation device in the same way as lambdas have been used before. They will make it far too easy to write obfuscated (perlish?) Python code even inadvertent. Regards, Peter -- Peter Funk, Oldenburger Str.86, D-27777 Ganderkesee, Germany, Fax:+49 4222950260 office: +49 421 20419-0 (ArtCom GmbH, Grazer Str.8, D-28359 Bremen, Germany) From mahler@cyc.com Tue Oct 16 08:51:32 2001 From: mahler@cyc.com (Daniel Mahler) Date: Tue, 16 Oct 2001 02:51:32 -0500 Subject: [Python-Dev] conditional expressions? In-Reply-To: <LNBBLJKPBEHFEDALKOLCIECPMBAA.tim.one@home.com> References: <15307.44095.944381.833422@mcallister.cyc.com> <LNBBLJKPBEHFEDALKOLCIECPMBAA.tim.one@home.com> Message-ID: <15307.59140.626077.69525@mcallister.cyc.com> Tim Peters writes: > [Daniel Mahler] > > Has an if(cond,exp1,exp2) syntax been considered? > > Not seriously, no. That looks like a function call, and function arguments > are always evaluated in Python, and that works against seeing this as a > short-circuiting construct: > > q = if y then x/y else copysign(INFINITY, x) I do not see a problem: almost all of the infix operators also use eager evaluation, but 2 use lazy evaluation, so operator syntax by itself does not imply or even suggest nonstandard evaluation order and there is no systematic syntactic convention to indicate it. You just have to remember that "and" and "or" are the only lazy operators (you might count lambda) out of ~35 (by my count). -- anyway, I had to check my reference card to see they were the only ones. I do not see why this is any more problematic for functional syntax. Since there is no generic way of indicating lazyness at the expression level (":" kind of indicates it at the statement level), people will just have to remember that conditionals are special no matter which syntax is chosen. OTOH conditionals are like functions in the syntactic sense (they take expressions as arguments and can be used anywhere expressions can) and in the mathematical sense (in the absence of side effects in the argument expressions the result is a mathematical function of the arguments). In lazy languages (Haskell,Miranda,Gofer) "if" is just a regular function. Hmmm, now that I think about it, I wonder why python makes a strong distinction between statements and expressions, but does not distinguish between procedures and functions? > > It would have the advantage that existing > > syntax sensitive editors would handle without change. > > I don't see any problem there (and am responsible for two context sensitive > Python edit gimmicks, and used to be responsible for a third -- all I expect > to need to do is get "then" at token level colored as keyword). My editor does more than highlighting. What about autoindentation if you split the line? Will one need a trailing "\" if when splitting across lines? -- with the if/then/else syntax, we will be splitting lines in expressions. [and deep down inside python does not really like that ;)]. I actually prefer to split even concise syntax, for readability, and python is friendlier to splitting functional expressions than to splitting operator expressions. Functional syntax would do the "right thing" automatically, even when nested, because it does not involve changing the syntax of the language. It just introduces lazy/call-by-name semantics. [Guido, can we have user defineable call-by-name functions -- pleeeease ;)] Actually, I lied little -- nested if(_,_,_) confuses emacs python-mode, but change the name to, say, ifTE (short for ifThenElse) and it autoindents beautyfully x = ifTE(a, ifTE(b, x, y), ifTE(c, w, z) ) > > I think conditional exprs badly needed, > > but the if/then/else syntax seems too verbose for use inside > > expressions. > > Good -- then you won't be tempted to abuse it <0.5 wink>. But one should be tempted to *use* them, otherwise one can just drop the feature, preventing abuse entirely. Note that "z = if a then x else y" is 2/3 syntactic sugar. I can see that "z = a ? x : y" is somewhat cryptic but no more than "x >> y", "x % y" or "x ^ y" ("^" is exponentiation in most other languages). Of course, "z = if(a,x,y)" is just ... ;). Daniel > best-in-small-doses-ly y'rs - tim > > > _______________________________________________ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev From paul@ActiveState.com Tue Oct 16 11:13:35 2001 From: paul@ActiveState.com (Paul Prescod) Date: Tue, 16 Oct 2001 03:13:35 -0700 Subject: [Python-Dev] conditional expressions? References: <LNBBLJKPBEHFEDALKOLCMEDAMBAA.tim.one@home.com> Message-ID: <3BCC084F.FCBB2CF0@ActiveState.com> Tim Peters wrote: > > ... > > I was its only fan within PythonLabs, and I was lukewarm. The trick is to > view "if c else" as a new short-circuiting infix binary operator (for each > c), and then it makes great-- even elegant --sense. But it's apparently a > major strain to see it that way. It had at least one sworn mortal enemy, > and I think Fred Drake was so upset at the idea that he couldn't speak. It actually read more easily to me the first time. As I said on python-list: unit is meters if metric, else feet versus unit is, if metric, meters, else feet At first I didn't like that it split the two options but I think that that actually helps you to keep clear in your head what the "default" is and what the "alternative" is. Paul Prescod From gward@python.net Tue Oct 16 14:00:58 2001 From: gward@python.net (Greg Ward) Date: Tue, 16 Oct 2001 09:00:58 -0400 Subject: [Python-Dev] conditional expressions? In-Reply-To: <3BCC084F.FCBB2CF0@ActiveState.com> References: <LNBBLJKPBEHFEDALKOLCMEDAMBAA.tim.one@home.com> <3BCC084F.FCBB2CF0@ActiveState.com> Message-ID: <20011016090058.A2546@cthulhu.mems-exchange.org> On 16 October 2001, Paul Prescod said: > It actually read more easily to me the first time. As I said on > python-list: > > unit is meters if metric, else feet > > versus > > unit is, if metric, meters, else feet That's confusing Python with English. They're two different languages, with different syntaxes, different semantics, and very different purposes. I wonder if a native speaker of German or Cantonese would agree with you? I really don't see what the big fuss is about. Algol got it right 40 years ago with "if x then y else z", and (nearly?) every language since then has either made conditional expressions painfully obfuscated (Lisp, C and its ilk) or forgotten them entirely (Python). They're very useful, and just as easily abused as other powerful constructs like list comprehensions, regular expressions, or lambdas. But just because something can be abused doesn't mean it should be banned. Down that road lies Nanny Python. Greg -- Greg Ward - geek gward@python.net http://starship.python.net/~gward/ Vote Cthulhu -- why settle for a lesser evil? From guido@python.org Tue Oct 16 14:17:37 2001 From: guido@python.org (Guido van Rossum) Date: Tue, 16 Oct 2001 09:17:37 -0400 Subject: [Python-Dev] conditional expressions? In-Reply-To: Your message of "Tue, 16 Oct 2001 01:15:05 EDT." <LNBBLJKPBEHFEDALKOLCMEDAMBAA.tim.one@home.com> References: <LNBBLJKPBEHFEDALKOLCMEDAMBAA.tim.one@home.com> Message-ID: <200110161317.JAA07196@cj20424-a.reston1.va.home.com> > [Greg Ewing] > > Has the "x if c else y" idea been totally rejected? > > It doesn't require a new keyword, or parentheses to > > disambiguate from an if statement. > > I was its only fan within PythonLabs, and I was lukewarm. The trick is to > view "if c else" as a new short-circuiting infix binary operator (for each > c), and then it makes great-- even elegant --sense. But it's apparently a > major strain to see it that way. It had at least one sworn mortal enemy, > and I think Fred Drake was so upset at the idea that he couldn't speak. Also I believe it would create a parsing problem in list comprehensions; compare these two: [x for x in range(10) if prime(x)] # 'if' is part of list compreh. [x for x in range(10) if small else range(100)] # 'if-then' construct --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Tue Oct 16 14:24:40 2001 From: guido@python.org (Guido van Rossum) Date: Tue, 16 Oct 2001 09:24:40 -0400 Subject: [Python-Dev] conditional expressions? In-Reply-To: Your message of "Tue, 16 Oct 2001 02:51:32 CDT." <15307.59140.626077.69525@mcallister.cyc.com> References: <15307.44095.944381.833422@mcallister.cyc.com> <LNBBLJKPBEHFEDALKOLCIECPMBAA.tim.one@home.com> <15307.59140.626077.69525@mcallister.cyc.com> Message-ID: <200110161324.JAA07249@cj20424-a.reston1.va.home.com> Even after reading Daniel's passionate plea, if(cond,expr,expr) looks ugly to me. It also doesn't solve the parsing ambiguity problem -- it still can't be allowed in an expression that starts a line because the parser doesn't know whether it's an if expression or an if statement without many tokens of lookahead. So, -1 for if(cond,expr,expr). --Guido van Rossum (home page: http://www.python.org/~guido/) From walter@livinglogic.de Tue Oct 16 14:46:05 2001 From: walter@livinglogic.de (Walter =?ISO-8859-1?Q?D=F6rwald?=) Date: Tue, 16 Oct 2001 15:46:05 +0200 Subject: [Python-Dev] conditional expressions? References: <15307.44095.944381.833422@mcallister.cyc.com> <LNBBLJKPBEHFEDALKOLCIECPMBAA.tim.one@home.com> <15307.59140.626077.69525@mcallister.cyc.com> <200110161324.JAA07249@cj20424-a.reston1.va.home.com> Message-ID: <3BCC3A1D.8000308@livinglogic.de> Guido van Rossum wrote: > Even after reading Daniel's passionate plea, if(cond,expr,expr) looks > ugly to me. It also doesn't solve the parsing ambiguity problem -- it > still can't be allowed in an expression that starts a line because the > parser doesn't know whether it's an if expression or an if statement > without many tokens of lookahead. This can be solved by simply choosing a different name: iif(cond, expr, expr) when(cond, expr, expr) cond(cond, expr, expr) check(cond, expr, expr) ... > So, -1 for if(cond,expr,expr). Bye, Walter D=F6rwald From guido@python.org Tue Oct 16 15:20:59 2001 From: guido@python.org (Guido van Rossum) Date: Tue, 16 Oct 2001 10:20:59 -0400 Subject: [Python-Dev] conditional expressions? In-Reply-To: Your message of "Tue, 16 Oct 2001 15:46:05 +0200." <3BCC3A1D.8000308@livinglogic.de> References: <15307.44095.944381.833422@mcallister.cyc.com> <LNBBLJKPBEHFEDALKOLCIECPMBAA.tim.one@home.com> <15307.59140.626077.69525@mcallister.cyc.com> <200110161324.JAA07249@cj20424-a.reston1.va.home.com> <3BCC3A1D.8000308@livinglogic.de> Message-ID: <200110161420.KAA07533@cj20424-a.reston1.va.home.com> > This can be solved by simply choosing a different name: > iif(cond, expr, expr) > when(cond, expr, expr) > cond(cond, expr, expr) > check(cond, expr, expr) > ... -1 too. (1) In order to do short-circuiting evaluation, the name would have to be recognized by the parser, which means it would have to be a new keyword. None of these keywords sound attractive. (2) It looks too much like a function call for my comfort. I still like (if cond then expr else expr) better than any of these. --Guido van Rossum (home page: http://www.python.org/~guido/) From paul@pfdubois.com Tue Oct 16 16:49:13 2001 From: paul@pfdubois.com (Paul F. Dubois) Date: Tue, 16 Oct 2001 08:49:13 -0700 Subject: [Python-Dev] Conditional expressions and sequences Message-ID: <000201c1565a$2700e520$3d01a8c0@plstn1.sfba.home.com> One of the most cherished of Numpy operations is: x = where (condition, t, f) which creates an array x, shaped like condition. Each element of this result is set to the corresponding element of t or f depending on whether the corresponding element of condition is true or false. Either or both of t and f can be a scalar but otherwise must be the same shape as condition. The reason this is cherished is, of course, that it avoids a scalar loop with an if-test in it. Scalar loops make Nummies cringe. Now in your new syntax this is essentially x = if condition then t else f Therefore it would be cool if the implementation was x=condition.__where__(t,f). This slot could be overwritten by classes like Numeric's array. If condition was a list this might be useful too. Never mind how I spelled __where__, please. The naming would require thought. Ducking-down-behind-the-barricade-'ly yours, Paul P.S. I once learned a language where you could say x = y if c and it was yummy. From skip@pobox.com (Skip Montanaro) Tue Oct 16 17:04:51 2001 From: skip@pobox.com (Skip Montanaro) (Skip Montanaro) Date: Tue, 16 Oct 2001 11:04:51 -0500 Subject: [Python-Dev] Conditional expressions and sequences In-Reply-To: <000201c1565a$2700e520$3d01a8c0@plstn1.sfba.home.com> References: <000201c1565a$2700e520$3d01a8c0@plstn1.sfba.home.com> Message-ID: <15308.23203.242343.613940@beluga.mojam.com> Paul> Now in your new syntax this is essentially Paul> x = if condition then t else f Paul> Therefore it would be cool if the implementation was Paul> x=condition.__where__(t,f). This slot could be overwritten by Paul> classes like Numeric's array. If condition was a list this might Paul> be useful too. Unfortunately, that requires both t and f to be evaluated. In a NumPy setting that's probably not a big deal because in most cases both values will be used in the resulting output. In a scalar context, the short-circuit nature of the expression is often important. If implemented as any kind of function you've lost that capability. -- Skip Montanaro (skip@pobox.com) http://www.mojam.com/ http://www.musi-cal.com/ From mahler@cyc.com Tue Oct 16 17:11:06 2001 From: mahler@cyc.com (Daniel Mahler) Date: Tue, 16 Oct 2001 11:11:06 -0500 Subject: [Python-Dev] conditional expressions? In-Reply-To: <200110161420.KAA07533@cj20424-a.reston1.va.home.com> References: <15307.44095.944381.833422@mcallister.cyc.com> <LNBBLJKPBEHFEDALKOLCIECPMBAA.tim.one@home.com> <15307.59140.626077.69525@mcallister.cyc.com> <200110161324.JAA07249@cj20424-a.reston1.va.home.com> <3BCC3A1D.8000308@livinglogic.de> <200110161420.KAA07533@cj20424-a.reston1.va.home.com> Message-ID: <15308.23578.120938.316757@mcallister.cyc.com> Guido van Rossum writes: > (1) In order to do short-circuiting evaluation, the name would have to > be recognized by the parser, which means it would have to be a new > keyword. That surprises me. In a conventional compiler architecture, I would expect this to be a static analysis and code generation issue. In languages that have multiple parameter passing disciplines (eg Pascal, Simula), the parser does not track how any given argument is being passed. > (2) It looks too much like a function call for my comfort. I am trying to make the case that it *is* a function call, with args 2 & 3 being passed by-name. Or at least it can very sensibly be viewed and implemented that way. > > I still like > > (if cond then expr else expr) > > better than any of these. :( > --Guido van Rossum (home page: http://www.python.org/~guido/) > > _______________________________________________ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev Daniel From guido@python.org Tue Oct 16 17:24:43 2001 From: guido@python.org (Guido van Rossum) Date: Tue, 16 Oct 2001 12:24:43 -0400 Subject: [Python-Dev] conditional expressions? In-Reply-To: Your message of "Tue, 16 Oct 2001 11:11:06 CDT." <15308.23578.120938.316757@mcallister.cyc.com> References: <15307.44095.944381.833422@mcallister.cyc.com> <LNBBLJKPBEHFEDALKOLCIECPMBAA.tim.one@home.com> <15307.59140.626077.69525@mcallister.cyc.com> <200110161324.JAA07249@cj20424-a.reston1.va.home.com> <3BCC3A1D.8000308@livinglogic.de> <200110161420.KAA07533@cj20424-a.reston1.va.home.com> <15308.23578.120938.316757@mcallister.cyc.com> Message-ID: <200110161624.f9GGOhI19035@odiug.digicool.com> > > (1) In order to do short-circuiting evaluation, the name would have to > > be recognized by the parser, which means it would have to be a new > > keyword. > > That surprises me. > In a conventional compiler architecture, > I would expect this to be a static analysis and code generation issue. > In languages that have multiple parameter passing disciplines > (eg Pascal, Simula), > the parser does not track how any given argument is being passed. This just shows how little you know about how Python is implemented. ;-( > > (2) It looks too much like a function call for my comfort. > > I am trying to make the case that it *is* a function call, > with args 2 & 3 being passed by-name. > Or at least it can very sensibly be viewed and implemented that way. Python has no by-name argument passing mechanisms, and adding one just for this purpose strikes me as the tail wagging the dog. --Guido van Rossum (home page: http://www.python.org/~guido/) From hamish_lawson@yahoo.co.uk Tue Oct 16 17:59:29 2001 From: hamish_lawson@yahoo.co.uk (=?iso-8859-1?q?Hamish=20Lawson?=) Date: Tue, 16 Oct 2001 17:59:29 +0100 (BST) Subject: [Python-Dev] What happened to Guido's plans for (S1; S2; ...; Sn; E) ? Message-ID: <20011016165929.54794.qmail@web11001.mail.yahoo.com> The current discussion on conditional expressions has at times touched on the ever-recurring issue of assignment in expressions. It reminded me that a proposal by Kevin Digweed back in early 1999 had actually managed to obtain Guido's approval as a reasonably Pythonic approach to allowing assignment in expressions, and the proposal seemed intended to make its eventual way into Python. Is this still the case? Kevin's original proposal: | What about allowing an optional "assignment only" suite | between "while" and the test condition. This suite will be executed | before testing the condition each time | <http://groups.google.com/groups?hl=en&selm=78naok%242ed%241%40nnrp1.dejanews.com> Guido then expanded on the proposal and seemed to give it his approval: | Why stop there? It could easily be any number of "small" statements | separated by semicolons with the restriction that the final one must | be an expression and that the ones before it must not be flow-control | statements (return, raise, continue, break). Other useful statements | to allow are print (for debugging) and assert, and for all I care | import, del, pass, and exec. | | Note that I like this for 2.0, not so much for for 1.6 (if only | because I don't want 1.6 to make existing books obsolete). | <http://groups.google.com/groups?hl=en&selm=199902021501.KAA07333%40eric.cnri.reston.va.us> Guido also clarified its intended syntax and semantics: | In my interpretation, the expression (S1; S2; ...; Sn; E) yields the | value of expression E -- S1-Sn are only there for their side effects. | <http://groups.google.com/groups?hl=en&selm=199902031343.IAA09480%40eric.cnri.reston.va.us> Hamish Lawson ____________________________________________________________ Do You Yahoo!? Get your free @yahoo.co.uk address at http://mail.yahoo.co.uk or your free @yahoo.ie address at http://mail.yahoo.ie From mahler@cyc.com Tue Oct 16 18:02:00 2001 From: mahler@cyc.com (Daniel Mahler) Date: Tue, 16 Oct 2001 12:02:00 -0500 Subject: [Python-Dev] conditional expressions? In-Reply-To: <200110161624.f9GGOhI19035@odiug.digicool.com> References: <15307.44095.944381.833422@mcallister.cyc.com> <LNBBLJKPBEHFEDALKOLCIECPMBAA.tim.one@home.com> <15307.59140.626077.69525@mcallister.cyc.com> <200110161324.JAA07249@cj20424-a.reston1.va.home.com> <3BCC3A1D.8000308@livinglogic.de> <200110161420.KAA07533@cj20424-a.reston1.va.home.com> <15308.23578.120938.316757@mcallister.cyc.com> <200110161624.f9GGOhI19035@odiug.digicool.com> Message-ID: <15308.26632.664809.870977@mcallister.cyc.com> Guido van Rossum writes: > > > (1) In order to do short-circuiting evaluation, the name would have to > > > be recognized by the parser, which means it would have to be a new > > > keyword. > > > > That surprises me. > > In a conventional compiler architecture, > > I would expect this to be a static analysis and code generation issue. > > In languages that have multiple parameter passing disciplines > > (eg Pascal, Simula), > > the parser does not track how any given argument is being passed. > > This just shows how little you know about how Python is implemented. ;-( True, but I am learning. > > > > (2) It looks too much like a function call for my comfort. > > > > I am trying to make the case that it *is* a function call, > > with args 2 & 3 being passed by-name. > > Or at least it can very sensibly be viewed and implemented that way. > > Python has no by-name argument passing mechanisms, and adding one just > for this purpose strikes me as the tail wagging the dog. But it need not be the only one ;) > --Guido van Rossum (home page: http://www.python.org/~guido/) regards Daniel From paul@ActiveState.com Tue Oct 16 20:12:40 2001 From: paul@ActiveState.com (Paul Prescod) Date: Tue, 16 Oct 2001 12:12:40 -0700 Subject: [Python-Dev] Conditional expressions and sequences References: <000201c1565a$2700e520$3d01a8c0@plstn1.sfba.home.com> <15308.23203.242343.613940@beluga.mojam.com> Message-ID: <3BCC86A8.1CD4B229@ActiveState.com> Skip Montanaro wrote: > >... > > Unfortunately, that requires both t and f to be evaluated. Perhaps not. What if the __where__ function was passed two callable objects representing closures? if x then y else z -> x.__where__(lambda: y, lambda: z) I'm not saying I'm necessarily in favor of __where__. It is a kind of innovative and I haven't thought through all of the issues. Paul Prescod From greg@cosc.canterbury.ac.nz Wed Oct 17 01:51:39 2001 From: greg@cosc.canterbury.ac.nz (Greg Ewing) Date: Wed, 17 Oct 2001 13:51:39 +1300 (NZDT) Subject: [Python-Dev] conditional expressions? In-Reply-To: <200110161317.JAA07196@cj20424-a.reston1.va.home.com> Message-ID: <200110170051.NAA07586@s454.cosc.canterbury.ac.nz> Guido: > Someone else: > > > I was its only fan within PythonLabs, and I was lukewarm. The trick is to > > view "if c else" as a new short-circuiting infix binary operator (for each > > c), and then it makes great-- even elegant --sense. I think it makes sense in its own right -- no need for "tricks". > Also I believe it would create a parsing problem in list > comprehensions; > > [x for x in range(10) if small else range(100)] Parentheses would need to be required in that case: [x for x in (range(10) if small else range(100))] I don't see that as a big problem -- wanting to put a conditional expression in that position is going to be fairly rare. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From neal@metaslash.com Wed Oct 17 01:57:17 2001 From: neal@metaslash.com (Neal Norwitz) Date: Tue, 16 Oct 2001 20:57:17 -0400 Subject: [Python-Dev] more pychecker warnings from python-current Message-ID: <3BCCD76D.D1645A1D@metaslash.com> Well, almost python-current. The warnings were from a few days ago. Neal -- cgi.py:880: Local variable (traceback) not used copy_reg.py:47: self is argument in function not sure if this is really a problem or intended copy_reg.py:65: Function return types are inconsistent line 63 returns tuple of 3, but line 65, returns tuple of 2 doctest.py:528: Parameter (prefix) not used docstring says prefix is used, but it isn't locale.py:358: Redefining attribute (setlocale) original line (68) markupbase.py:31: No class attribute (rawdata) found markupbase.py:49: Local variable (sys) not used markupbase.py:71: No class attribute (unknown_decl) found markupbase.py:86: No class attribute (error) found markupbase.py:118: No class attribute (parse_comment) found markupbase.py:166: Local variable (n) not used mhlib.py:651: Local variable (messages) not used looks like a return in the if clause is necessary, otherwise exception pickle.py:607: No class attribute (persistent_load) found pickle.py:616: No class attribute (persistent_load) found profile.py:356: Local variable (a) not used on next line, return is not necessary, should a be returned? profile.py:368: Local variable (a) not used xmlrpclib.py:1: Imported module (sys) not used From guido@python.org Wed Oct 17 06:57:26 2001 From: guido@python.org (Guido van Rossum) Date: Wed, 17 Oct 2001 01:57:26 -0400 Subject: [Python-Dev] more pychecker warnings from python-current In-Reply-To: Your message of "Tue, 16 Oct 2001 20:57:17 EDT." <3BCCD76D.D1645A1D@metaslash.com> References: <3BCCD76D.D1645A1D@metaslash.com> Message-ID: <200110170557.BAA09026@cj20424-a.reston1.va.home.com> > cgi.py:880: Local variable (traceback) not used I believe this was to avoid a nasty problem when the 'import traceback' in print_exception() actually has to load the traceback module (as opposed to finding it already loaded in sys.modules); this could lose the pending traceback. I believe that problem is solved now so this can safely be removed. > copy_reg.py:47: self is argument in function > not sure if this is really a problem or intended This is intended -- _reduce() is called by the C method object.__reduce__ to do the heavy lifing that I didn't want to code in C (yet). > copy_reg.py:65: Function return types are inconsistent > line 63 returns tuple of 3, but line 65, returns tuple of 2 This is unfortunately the spec of the __reduce__ function -- it can return a 2-tuple or a 3-tuple. > doctest.py:528: Parameter (prefix) not used > docstring says prefix is used, but it isn't Tim? > locale.py:358: Redefining attribute (setlocale) original line (68) This is intentional. It confused me too. Note that the previous locale is stored in _setlocale -- there's a comment about that there. Still, it's all very confusing. Martin? > markupbase.py:31: No class attribute (rawdata) found > markupbase.py:49: Local variable (sys) not used > markupbase.py:71: No class attribute (unknown_decl) found > markupbase.py:86: No class attribute (error) found > markupbase.py:118: No class attribute (parse_comment) found > markupbase.py:166: Local variable (n) not used This is an abstract base class -- these are references to instance variables of a derived class. Except for the two "local variable not used" warnings -- these are relics from the code refactoring that created the abstract class. Looks like a job for Fred. > mhlib.py:651: Local variable (messages) not used > looks like a return in the if clause is necessary, otherwise exception No, this is done for the side effect of setting self.last (mentioned in the comment in listmessages()). I've fixed this by getting rid of the "messages =" part. > pickle.py:607: No class attribute (persistent_load) found > pickle.py:616: No class attribute (persistent_load) found AFAIK this is intentional; a derived class would have to define persistent_load() for this feature to work. > profile.py:356: Local variable (a) not used > on next line, return is not necessary, should a be returned? > profile.py:368: Local variable (a) not used Tim? > xmlrpclib.py:1: Imported module (sys) not used Fred already fixed this one. --Guido van Rossum (home page: http://www.python.org/~guido/) From fdrake@acm.org Wed Oct 17 14:26:43 2001 From: fdrake@acm.org (Fred L. Drake, Jr.) Date: Wed, 17 Oct 2001 09:26:43 -0400 Subject: [Python-Dev] more pychecker warnings from python-current In-Reply-To: <200110170557.BAA09026@cj20424-a.reston1.va.home.com> References: <3BCCD76D.D1645A1D@metaslash.com> <200110170557.BAA09026@cj20424-a.reston1.va.home.com> Message-ID: <15309.34579.733687.799625@grendel.zope.com> Guido van Rossum writes: > > markupbase.py:31: No class attribute (rawdata) found > > markupbase.py:49: Local variable (sys) not used > > markupbase.py:71: No class attribute (unknown_decl) found > > markupbase.py:86: No class attribute (error) found > > markupbase.py:118: No class attribute (parse_comment) found > > markupbase.py:166: Local variable (n) not used > > This is an abstract base class -- these are references to instance > variables of a derived class. Except for the two "local variable not > used" warnings -- these are relics from the code refactoring that > created the abstract class. Looks like a job for Fred. I expect to check something in for this tonight. > > profile.py:356: Local variable (a) not used > > on next line, return is not necessary, should a be returned? > > profile.py:368: Local variable (a) not used > > Tim? Sorry; I nailed this one already. ;-) -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> PythonLabs at Zope Corporation From neal@metaslash.com Wed Oct 17 15:04:36 2001 From: neal@metaslash.com (Neal Norwitz) Date: Wed, 17 Oct 2001 10:04:36 -0400 Subject: [Python-Dev] bug or feature in imp.load_module()? Message-ID: <3BCD8FF4.96457AD8@metaslash.com> module = imp.load_module(path, file, filename, smt) After returning from imp.load_module(), the file paramter passed in may be at the beginning or end depending on whether it was byte compiled. If the file was already compiled, the file returned will stay at the beginning. If the file was byte compiled in load_module(), the file will be at the end. The following program may demonstrate better: ##################### import imp, sys, os FILE_BASE = 'x' PY_FILE = FILE_BASE + '.py' def load(f): file, filename, smt = imp.find_module(f, sys.path) imp.load_module('', file, filename, smt) print 'File is at byte:', file.tell() def main(): try: os.unlink(PY_FILE + 'c') except: pass if not os.path.exists(PY_FILE): file = open(PY_FILE, 'w') file.close() load(FILE_BASE) load(FILE_BASE) if __name__ == '__main__': main() ##################### file sometimes being modified was unexpected. And I couldn't find the behaviour documented anywhere. This behaviour goes back to Python 1.5 at least AFAIR. Neal From Greg.Wilson@baltimore.com Wed Oct 17 15:46:45 2001 From: Greg.Wilson@baltimore.com (Greg Wilson) Date: Wed, 17 Oct 2001 10:46:45 -0400 Subject: [Python-Dev] re: conditional expressions Message-ID: <930BBCA4CEBBD411BE6500508BB3328F4322C4@nsamcanms1.ca.baltimore.com> I just showed the proposed conditional expression syntax to two colleagues, and asked them how they would write the "sign" function (-1 for negative, 0 for zero, +1 for positive) using it. Both of them wrote: s = if x < 0 then -1 elif x == 0 then 0 else 1 (Well, OK, one of them tested negative, then positive, then returned zero as the 'else', but that's a quibble.) I think most users will expect 'elif' to work in expressions, just as it does in statements; is this already part of the new syntax? Thanks, Greg ----------------------------------------------------------------------------------------------------------------- The information contained in this message is confidential and is intended for the addressee(s) only. If you have received this message in error or there are any problems please notify the originator immediately. The unauthorized use, disclosure, copying or alteration of this message is strictly forbidden. Baltimore Technologies plc will not be liable for direct, special, indirect or consequential damages arising from alteration of the contents of this message by a third party or as a result of any virus being passed on. In addition, certain Marketing collateral may be added from time to time to promote Baltimore Technologies products, services, Global e-Security or appearance at trade shows and conferences. This footnote confirms that this email message has been swept by Baltimore MIMEsweeper for Content Security threats, including computer viruses. From fredrik@pythonware.com Wed Oct 17 16:03:04 2001 From: fredrik@pythonware.com (Fredrik Lundh) Date: Wed, 17 Oct 2001 17:03:04 +0200 Subject: [Python-Dev] conditional expressions? References: <15307.44095.944381.833422@mcallister.cyc.com> <LNBBLJKPBEHFEDALKOLCIECPMBAA.tim.one@home.com> <15307.59140.626077.69525@mcallister.cyc.com> <200110161324.JAA07249@cj20424-a.reston1.va.home.com> <3BCC3A1D.8000308@livinglogic.de> <200110161420.KAA07533@cj20424-a.reston1.va.home.com> Message-ID: <007f01c1571c$d43c7fc0$ced241d5@hagrid> guido wrote: > I still like > > (if cond then expr else expr) > > better than any of these. I guess I've lost track of all the proposals here; what's the current main proposal? (if cond then expr else expr) with mandatory parentheses? not too bad -- but I doubt it will save me much time in real life... (probably a net loss: even if I could use this from time to time, it's not like I have to use much brain- power to spell it out in today's python...) </F> From barry@zope.com Wed Oct 17 18:27:59 2001 From: barry@zope.com (Barry A. Warsaw) Date: Wed, 17 Oct 2001 13:27:59 -0400 Subject: [Python-Dev] conditional expressions? References: <15307.44095.944381.833422@mcallister.cyc.com> <LNBBLJKPBEHFEDALKOLCIECPMBAA.tim.one@home.com> <15307.59140.626077.69525@mcallister.cyc.com> <200110161324.JAA07249@cj20424-a.reston1.va.home.com> <3BCC3A1D.8000308@livinglogic.de> <200110161420.KAA07533@cj20424-a.reston1.va.home.com> <007f01c1571c$d43c7fc0$ced241d5@hagrid> Message-ID: <15309.49055.970876.74880@anthem.wooz.org> >>>>> "FL" == Fredrik Lundh <fredrik@pythonware.com> writes: FL> I guess I've lost track of all the proposals here; what's FL> the current main proposal? Another good reason why there should be a PEP first. We talked about it yesterday and decided against trying to get anything like this into Py2.2. PEPs for 2.3 would be welcome. ;) -Barry From guido@python.org Wed Oct 17 18:27:04 2001 From: guido@python.org (Guido van Rossum) Date: Wed, 17 Oct 2001 13:27:04 -0400 Subject: [Python-Dev] conditional expressions? In-Reply-To: Your message of "Wed, 17 Oct 2001 17:03:04 +0200." <007f01c1571c$d43c7fc0$ced241d5@hagrid> References: <15307.44095.944381.833422@mcallister.cyc.com> <LNBBLJKPBEHFEDALKOLCIECPMBAA.tim.one@home.com> <15307.59140.626077.69525@mcallister.cyc.com> <200110161324.JAA07249@cj20424-a.reston1.va.home.com> <3BCC3A1D.8000308@livinglogic.de> <200110161420.KAA07533@cj20424-a.reston1.va.home.com> <007f01c1571c$d43c7fc0$ced241d5@hagrid> Message-ID: <200110171727.NAA16770@cj20424-a.reston1.va.home.com> > I guess I've lost track of all the proposals here; what's > the current main proposal? > > (if cond then expr else expr) > > with mandatory parentheses? Almost -- the parentheses can be left out in some cases, e.g. x = if 1 then 2 else 3 I agree with Greg Wilson that elif should be added too. > not too bad -- but I doubt it will save me much time in > real life... (probably a net loss: even if I could use this > from time to time, it's not like I have to use much brain- > power to spell it out in today's python...) That's my conclusion too. I went looking through the stdandard library for places that could use it and basically didn't find any in the 15 minutes before I got bored. So the patch is still there but this won't make it into 2.2. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Wed Oct 17 18:32:00 2001 From: guido@python.org (Guido van Rossum) Date: Wed, 17 Oct 2001 13:32:00 -0400 Subject: [Python-Dev] bug or feature in imp.load_module()? In-Reply-To: Your message of "Wed, 17 Oct 2001 10:04:36 EDT." <3BCD8FF4.96457AD8@metaslash.com> References: <3BCD8FF4.96457AD8@metaslash.com> Message-ID: <200110171732.NAA16813@cj20424-a.reston1.va.home.com> > module = imp.load_module(path, file, filename, smt) > > After returning from imp.load_module(), the file paramter passed in > may be at the beginning or end depending on whether it was byte > compiled. If the file was already compiled, the file returned will > stay at the beginning. If the file was byte compiled in > load_module(), the file will be at the end. This certainly doesn't surprise me -- in one case it read the source and in the other case it didn't read the source. If you pass a file to a function, I don't think that there is any guarantee to the file position when it returns, so I don't think this should be documented. I can't think of why you'd expect it to be at any specific position. --Guido van Rossum (home page: http://www.python.org/~guido/) From neal@metaslash.com Wed Oct 17 19:22:43 2001 From: neal@metaslash.com (Neal Norwitz) Date: Wed, 17 Oct 2001 14:22:43 -0400 Subject: [Python-Dev] bug or feature in imp.load_module()? References: <3BCD8FF4.96457AD8@metaslash.com> <200110171732.NAA16813@cj20424-a.reston1.va.home.com> Message-ID: <3BCDCC73.A516C31E@metaslash.com> Guido van Rossum wrote: > > > module = imp.load_module(path, file, filename, smt) > > > > After returning from imp.load_module(), the file paramter passed in > > may be at the beginning or end depending on whether it was byte > > compiled. If the file was already compiled, the file returned will > > stay at the beginning. If the file was byte compiled in > > load_module(), the file will be at the end. > > This certainly doesn't surprise me -- in one case it read the source > and in the other case it didn't read the source. > > If you pass a file to a function, I don't think that there is any > guarantee to the file position when it returns, so I don't think this > should be documented. I can't think of why you'd expect it to be at > any specific position. I'm not sure either. :-) It was odd for me to pass both file & filename. The doc says that: The file argument is an open file, and filename is the corresponding file name; these can be None and '', respectively, when the module is not being loaded from a file. I'm not sure how the module could not be loaded from a file (unless the implication is that the file argument is None and a new file will be opened from the filename). If the file can be None anytime, it makes more sense to me. But the last phrase in the doc seems to indicate the file argument should not be None. Perhaps, it's just me, it wouldn't be the first time. :-) Neal From guido@python.org Wed Oct 17 20:05:35 2001 From: guido@python.org (Guido van Rossum) Date: Wed, 17 Oct 2001 15:05:35 -0400 Subject: [Python-Dev] bug or feature in imp.load_module()? In-Reply-To: Your message of "Wed, 17 Oct 2001 14:22:43 EDT." <3BCDCC73.A516C31E@metaslash.com> References: <3BCD8FF4.96457AD8@metaslash.com> <200110171732.NAA16813@cj20424-a.reston1.va.home.com> <3BCDCC73.A516C31E@metaslash.com> Message-ID: <200110171905.PAA17035@cj20424-a.reston1.va.home.com> > > If you pass a file to a function, I don't think that there is any > > guarantee to the file position when it returns, so I don't think this > > should be documented. I can't think of why you'd expect it to be at > > any specific position. > > I'm not sure either. :-) It was odd for me to pass both file & filename. Well, you've got to realize that this is a really low-level interface designed to make certain optimizations possible. The filename is needed (a) for diagnostics, (b) for opening the .pyc file; the open file is a side effect from the search. It uses fopen() to check for file existence, to prevent silly things like existing files that cannot be opened for whatever reason). You don't want to have to open the file twice! > The doc says that: > > The file argument is an open file, and filename is > the corresponding file name; these can be None and '', > respectively, when the module is not being loaded from a file. > > I'm not sure how the module could not be loaded from a file (unless > the implication is that the file argument is None and a new file will > be opened from the filename). Read the C source and weep. :-) Some systems have other places from which they can load modules, e.g. on the Mac a module can be in a "resource". Also there are frozen modules. > If the file can be None anytime, it makes more sense to me. > But the last phrase in the doc seems to indicate the file argument > should not be None. > > Perhaps, it's just me, it wouldn't be the first time. :-) Why are you trying to use this? Can't you use __import__? --Guido van Rossum (home page: http://www.python.org/~guido/) From neal@metaslash.com Wed Oct 17 20:24:05 2001 From: neal@metaslash.com (Neal Norwitz) Date: Wed, 17 Oct 2001 15:24:05 -0400 Subject: [Python-Dev] bug or feature in imp.load_module()? References: <3BCD8FF4.96457AD8@metaslash.com> <200110171732.NAA16813@cj20424-a.reston1.va.home.com> <3BCDCC73.A516C31E@metaslash.com> <200110171905.PAA17035@cj20424-a.reston1.va.home.com> Message-ID: <3BCDDAD5.EAEEE1E4@metaslash.com> Guido van Rossum wrote: [stuff about imp.load_module() deleted] > Why are you trying to use this? Can't you use __import__? This was from ancient code. I think __import__ could be used. I haven't modified this code for a long time (I don't even think I wrote the original code). The code is used to load the modules so they can be processed by pychecker. When I change pychecker to not actually import the file, using imp.load_module() or __import__ will be moot. This was just something I noticed and seemed inconsistent to me. Although it was easy enough to overcome with a file.seek(0). Neal From tim.one@home.com Thu Oct 18 03:08:03 2001 From: tim.one@home.com (Tim Peters) Date: Wed, 17 Oct 2001 22:08:03 -0400 Subject: [Python-Dev] re: conditional expressions In-Reply-To: <930BBCA4CEBBD411BE6500508BB3328F4322C4@nsamcanms1.ca.baltimore.com> Message-ID: <LNBBLJKPBEHFEDALKOLCKEJIMBAA.tim.one@home.com> [Greg Wilson] > I just showed the proposed conditional expression syntax > to two colleagues, and asked them how they would write > the "sign" function (-1 for negative, 0 for zero, +1 for > positive) using it. Both of them wrote: > > s = if x < 0 then -1 elif x == 0 then 0 else 1 > > (Well, OK, one of them tested negative, then positive, then > returned zero as the 'else', but that's a quibble.) Quibble? A professor in college took points off one of my programs because I didn't test for 0 first. He explained that testing for either negative or positive first left you vulnerable on 1's-complement machines, where comparisons for <0 or >0 just checked the sign bit, and either form of 1's-comp 0 would pass one of those tests. I looked at him like he was insane. He looked at me the same way. He was old enough that I suppose he's dead now. Ha! I win <wink>. elif-is-file-spelled-backwards-ly y'rs - tim From tim.one@home.com Thu Oct 18 03:30:54 2001 From: tim.one@home.com (Tim Peters) Date: Wed, 17 Oct 2001 22:30:54 -0400 Subject: [Python-Dev] more pychecker warnings from python-current In-Reply-To: <200110170557.BAA09026@cj20424-a.reston1.va.home.com> Message-ID: <LNBBLJKPBEHFEDALKOLCGEJJMBAA.tim.one@home.com> >> doctest.py:528: Parameter (prefix) not used >> docstring says prefix is used, but it isn't > Tim? No, the docstring says prefix is ignored: def is_private(prefix, base): """prefix, base -> true iff name prefix + "." + base is "private". Prefix may be an empty string, and base does not contain a period. Prefix is ignored (although functions you write conforming to this ^^^^^^^^^^^^^^^^^ protocol may make use of it). ... is_private is a user-replaceable function, and doctest is careful to *pass* the correct prefix to it, but the default implementation makes no use of prefix. It's there for advanced users who want to factor in, e.g., the name of a containing class, when making an "is this name private?" decision. I wouldn't be averse to adding, e.g., if 0: prefix # make prefix appear used to checking tools but sooner or later checking tools will complain about that too. In my last job, we resorted to a C++ macro of the flavor #define UNUSED_VAR_OK(VAR) (void)ignore_var((void*)&(VAR)) to shut up compiler wngs about intentionally ignored vrbls, and actually had a do-nothing extern void ignore_var(void*) function -- nothing short of that shut up every compiler we used. can't-win-ly y'rs - tim From neal@metaslash.com Thu Oct 18 03:53:48 2001 From: neal@metaslash.com (Neal Norwitz) Date: Wed, 17 Oct 2001 22:53:48 -0400 Subject: [Python-Dev] more pychecker warnings from python-current References: <LNBBLJKPBEHFEDALKOLCGEJJMBAA.tim.one@home.com> Message-ID: <3BCE443C.E032A033@metaslash.com> Tim Peters wrote: > > >> doctest.py:528: Parameter (prefix) not used > >> docstring says prefix is used, but it isn't > > > Tim? > > No, the docstring says prefix is ignored: > > def is_private(prefix, base): > """prefix, base -> true iff name prefix + "." + base is "private". > > Prefix may be an empty string, and base does not contain a period. > Prefix is ignored (although functions you write conforming to this > ^^^^^^^^^^^^^^^^^ > protocol may make use of it). > ... Oops, sorry about that, I saw all the example usages in the docstring and thought it was used. I didn't pay close enough attention. > I wouldn't be averse to adding, e.g., > > if 0: > prefix # make prefix appear used to checking tools > > but sooner or later checking tools will complain about that too. I wouldn't worry about it. I think I will eventually create a suppressions dictionary for the std library. So any warning that checker might normally output will not produce a warning, when it has been determined the code is correct. The capability exists to do this today. However, my concern is putting in a suppression and later that suppression masks a real error. In general, we (collectively) can define best practices for both code and checking tools, at least pychecker. Neal From thomas.heller@ion-tof.com Thu Oct 18 09:28:58 2001 From: thomas.heller@ion-tof.com (Thomas Heller) Date: Thu, 18 Oct 2001 10:28:58 +0200 Subject: [Python-Dev] Function arguments Message-ID: <011d01c157ae$eff659b0$e000a8c0@thomasnotebook> Call me weird, but sometimes I need functions taking a number of positional arguments, some named arguments with default values, and other named arguments as well. Something like this (which does _not_ work): def function(*args, defarg1=None, defarg2=0, **kw): ... The usual way to program this is: def function(*args, ***kw): if kw.has_key('defarg1'): defarg1 = kw['defarg1'] del kw['defarg1'] else: defarg1 = None if kw.has_key('defarg2'): defarg1 = kw['defarg2'] del kw['defarg2'] else: defarg2 = 0 # ...process the positional arguments in args # ...process the remaining named arguments in kw Doesn't look very pretty IMO, and using the dictionaries' get method doesn't help. I was thinking of a popitem() dictionary method taking (optionally) 2 arguments: the name of the item to pop, and the default value to return if the item is not present in the dictionary: >>> d = {'a': 2, 'b': 3} >>> d.popitem('defarg', 0) 0 >>> d {'a': 2, 'b': 3} >>> d.popitem('a', 100) 2 >>> d {'b': 3} >>> Opinions? Thomas From mwh@python.net Thu Oct 18 10:42:31 2001 From: mwh@python.net (Michael Hudson) Date: 18 Oct 2001 05:42:31 -0400 Subject: [Python-Dev] more pychecker warnings from python-current In-Reply-To: "Tim Peters"'s message of "Wed, 17 Oct 2001 22:30:54 -0400" References: <LNBBLJKPBEHFEDALKOLCGEJJMBAA.tim.one@home.com> Message-ID: <2madypp9mg.fsf@starship.python.net> "Tim Peters" <tim.one@home.com> writes: > I wouldn't be averse to adding, e.g., > > if 0: > prefix # make prefix appear used to checking tools > > but sooner or later checking tools will complain about that too. If they work by scanning bytecode (as PyChecker does, doesn't it?), that won't work today. pointless-ly y'rs, M. -- Screaming 14-year-old boys attempting to prove to each other that they are more 3133t than j00. -- Reason #8 for quitting slashdot today, from http://www.cs.washington.edu/homes/klee/misc/slashdot.html From fdrake@acm.org Thu Oct 18 13:00:00 2001 From: fdrake@acm.org (Fred L. Drake, Jr.) Date: Thu, 18 Oct 2001 08:00:00 -0400 Subject: [Python-Dev] Function arguments In-Reply-To: <011d01c157ae$eff659b0$e000a8c0@thomasnotebook> References: <011d01c157ae$eff659b0$e000a8c0@thomasnotebook> Message-ID: <15310.50240.333969.995440@grendel.zope.com> Thomas Heller writes: > I was thinking of a popitem() dictionary method taking > (optionally) 2 arguments: the name of the item to pop, > and the default value to return if the item is not present A one-arg version of the {}.popitem() method was discussed extensively, but some people did not think it sufficiently useful to include that variation. (I don't think anyone suggested a 2-arg version at the time.) This certainly leads me to think the decision not to support a one-arg version should be considered, or maybe a two-arg version. However, I do expect the usage with args and without args are substantially different. Perhaps there should be a different method that has the one- and two-arg versions you describe, but does not support the 0-arg semantics of the current popitem()? The catch is that I'd name this popitem() as well. ;-) Anyone else? -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> PythonLabs at Zope Corporation From thomas.heller@ion-tof.com Thu Oct 18 13:27:46 2001 From: thomas.heller@ion-tof.com (Thomas Heller) Date: Thu, 18 Oct 2001 14:27:46 +0200 Subject: [Python-Dev] Function arguments References: <011d01c157ae$eff659b0$e000a8c0@thomasnotebook> <15310.50240.333969.995440@grendel.zope.com> Message-ID: <037501c157d0$4b1e3710$e000a8c0@thomasnotebook> > Thomas Heller writes: > > I was thinking of a popitem() dictionary method taking > > (optionally) 2 arguments: the name of the item to pop, > > and the default value to return if the item is not present > From: "Fred L. Drake, Jr." <fdrake@acm.org> > A one-arg version of the {}.popitem() method was discussed > extensively, but some people did not think it sufficiently useful to > include that variation. (I don't think anyone suggested a 2-arg > version at the time.) I tried to look it up in the archives. Although (because?) there are lots of messages about popitem() I cannot locate the relevant ones quick. > This certainly leads me to think the decision not to support a > one-arg version should be considered, or maybe a two-arg version. > However, I do expect the usage with args and without args are > substantially different. Perhaps there should be a different method > that has the one- and two-arg versions you describe, but does not > support the 0-arg semantics of the current popitem()? The catch is > that I'd name this popitem() as well. ;-) > Anyone else? After posting I noticed that {}.popitem() returns a (key, value) tuple. To make it absolutely clear (most of the time I'm not very lucky in explaining what I really want): What I mean is a {}.get(key, [value=None]) method, which returns the value, and removes the (key, value) pair from the dict if it is present. No idea how it should be named - since it does only return the value, popitem doesn't seem a good idea. Thomas From loewis@informatik.hu-berlin.de Thu Oct 18 15:28:42 2001 From: loewis@informatik.hu-berlin.de (Martin von Loewis) Date: Thu, 18 Oct 2001 16:28:42 +0200 (MEST) Subject: [Python-Dev] 2.1.1 maintenance branch Message-ID: <200110181428.QAA16670@paros.informatik.hu-berlin.de> How should the 2.1 branch be used? Is it acceptable to propagate patches from the mainline to that branch? Is there somebody that needs to review changes made to the branch? Specific case in question: Anthony Baxter found Zope crashing, and apparently ceval.c 2.277 fixes that crash. I would now like to forward this patch to the 2.1 branch. Regards, Martin From skip@pobox.com (Skip Montanaro) Thu Oct 18 15:29:10 2001 From: skip@pobox.com (Skip Montanaro) (Skip Montanaro) Date: Thu, 18 Oct 2001 09:29:10 -0500 Subject: [Python-Dev] Function arguments In-Reply-To: <15310.50240.333969.995440@grendel.zope.com> References: <011d01c157ae$eff659b0$e000a8c0@thomasnotebook> <15310.50240.333969.995440@grendel.zope.com> Message-ID: <15310.59190.519026.561865@beluga.mojam.com> It's perhaps worth pointing out that dicts already have a zero-arg popitem method. I don't think I would use Thomas's proposed popitem so frequently that the fairly obvious v = d.get(k, default) try: del d[k] except KeyError: pass would be all that much trouble to type. Those are the semantics you're proposing, right? Skip From nas@python.ca Thu Oct 18 15:38:15 2001 From: nas@python.ca (Neil Schemenauer) Date: Thu, 18 Oct 2001 07:38:15 -0700 Subject: [Python-Dev] python2.1.1 SEGV in GC on Solaris 2.7 In-Reply-To: <E15uASk-0004Ax-00@usw-sf-web3.sourceforge.net>; from noreply@sourceforge.net on Thu, Oct 18, 2001 at 03:31:42AM -0700 References: <E15uASk-0004Ax-00@usw-sf-web3.sourceforge.net> Message-ID: <20011018073815.A22506@glacier.arctrix.com> [I'm moving the discussion here from SF, using the tracker is too painful.] Anthony Baxter: > I've got a Zope installation where python2.1.1 is > segfaulting on Solaris2.7 - it's running a largish > ZEO server. [..] > Here's the trace with debugging enabled: > > #0 0xff00 in ?? () > #1 0x402f0 in collect (young=0x9b538, old=0x9b544) at > ./Modules/gcmodule.c:379 > #2 0x405a8 in collect_generations () at > ./Modules/gcmodule.c:484 > #3 0x40624 in _PyGC_Insert (op=0xbc1f24) at > ./Modules/gcmodule.c:507 > #4 0x5a224 in PyList_New (size=0) at Objects/listobject.c:61 > #5 0x21bc8 in eval_code2 (co=0x1cb370, globals=0x21bc0, > locals=0x67, > args=0x0, argcount=1, kws=0xf89b24, kwcount=0, defs=0x0, > defcount=0, > closure=0xbc1f24) at Python/ceval.c:1741 > > Next trick is to rebuild without any optimisation (sigh) > as I suspect that it's inlined subtract_refs(). Martin v. Löwis: > It would be interesting what the value of "gc" is at the > time of the crash. It looks like you got an object that > claims to support GC but has a null tp_traverse. Anthony Baxter: > Ok, I have an intact core file, and a matching binary, > no optimisations, nothing. This crash is showing the > crash at line 166 of gcmodule.c > traverse = PyObject_FROM_GC(gc)->ob_type->tp_traverse; > PyObject_FROM_GC(gc)->ob_type in this case is > > $24 = {ob_refcnt = 1, ob_type = 0x0} > > To check my logic, I checked gc_next and gc_prev using > the same GDB magic, and they correctly show up as a tuple > and an instance method. > > Some fiddling around seems to rule out stack space as the > problem, as well. We're going to try and see if purify > helps here, but the problem looks to be a junk object - > I have no idea how to track this down further. Help? > Would taking the horrible horrible hack of removing the > object from the gc linked list if ob_type is null help? > Well, it'd stop the crashes, anyway. Martin v. Löwis: > There are two options: > > a) the object isn't really a GC object, i.e. has no GC > header. In gdb, you can try to cast gc to PyObject*, then > see if the resulting pointer has a better ob_type (this is > unlikely, though, since the logic entering the object was > already using fromgc/togc) > > b) somebody has cleared the ob_type field. > > Can you guarantee that all extension modules have been > compiled with the 2.1.1 header files? > > Is the problem repeatable in the sense that gc will have the > same pointer value on each crash? If so, it is relatively > easy to track down: just set a gdb change watchpoint on the > address on the ob_type field of that address (note that > setting watchpoints is not possible until there is really > mapped memory on that address). > > If you can't analyse it through change breakpoints, I > recommend to annotate the interpreter in the following way: > in pyobject_init, put a printf that prints the address and > the tp_name of the type. In subtract_refs, if the ob_type > slot is null, print the address of the object and abort. > Then analyse the log to see whether a object really has been > allocated on that address, and what its type was (make sure > you consider the possibility that address are off by the > delta that FROM_GC adds). Anthony Baxter: > It's not a GC object. I'm positive all the extension > objects are correct - I just recompiled, without the > 1.5/2.0 headers around. > It's a different pointer each time round, unfortunately. It > also takes anything from 5 minutes to 2 hours to reproduce. > I've got about 4 copies of it running now, and I've got a > bunch of different core files. I've grabbed purify and an > eval license, and I'm feeding it the binary. > > The printf approach is probably not going to work - these > are busy busy Zope servers. Instead, my plan, assuming that > purify doesn't immediately spot a problem, is to change the > code so that if it gets a dud GC object, it will just bust > it out of the tree and let it leak, and print a message > saying so. Then I can quit the program, and purify will > tell me 'hey, you leaked!' and also tell me where it was > allocated. > > More concerning, about half the segfaults are not from the > GC at all, but from realloc in PyFrame_New (line 161 of > frameobject). These are the only two I'm getting - it's > split 50-50 amongst the 10 coredumps I have now. I'm not > sure whether to open a seperate bug for this. > > Has python2.1.1 been purified? With Zope and zope's > extensions? > > > Wow - it's amazing how this SF bug thing is so painful for > conversations :) The ob_type pointer must be getting cleared after the object has been added to the GC lists. The PyObject_IS_GC call in _PyGC_Insert would have segfaulted otherwise. Knowing the type of the object would be helpful in debugging the problem. I suggest reconsidering Martin's printf idea. You could add something like this to _PyGC_Insert: void _PyGC_Insert(PyObject *op) { static int did_open = 0; static FILE *log; if (!did_open) { did_open = 1; log = fopen("type.log", "w"); } fprintf(log, "%p %p\n", op, op->ob_type); ... Debugging this type of problem is really hard (as you already know) because the effect of the bug is found so far away from the source. Neil From guido@python.org Thu Oct 18 15:41:35 2001 From: guido@python.org (Guido van Rossum) Date: Thu, 18 Oct 2001 10:41:35 -0400 Subject: [Python-Dev] 2.1.1 maintenance branch In-Reply-To: Your message of "Thu, 18 Oct 2001 16:28:42 +0200." <200110181428.QAA16670@paros.informatik.hu-berlin.de> References: <200110181428.QAA16670@paros.informatik.hu-berlin.de> Message-ID: <200110181441.f9IEfZF02019@odiug.zope.com> > How should the 2.1 branch be used? Is it acceptable to propagate > patches from the mainline to that branch? Is there somebody that needs > to review changes made to the branch? > > Specific case in question: Anthony Baxter found Zope crashing, and > apparently ceval.c 2.277 fixes that crash. I would now like to forward > this patch to the 2.1 branch. I don't know that we're going to do a 2.1.2 release, but *if* we do, it will be released from the 2.1 branch, so it's a good idea to start migrating patches to that branch -- VERY selectively. This is a good one to start with: an obvious bug that causes a core dump. If someone (you?) volunteers to act as a 2.1.2 release manager, chances of a 2.1.2 release go way up. --Guido van Rossum (home page: http://www.python.org/~guido/) From Anthony Baxter <anthony@interlink.com.au> Thu Oct 18 16:11:20 2001 From: Anthony Baxter <anthony@interlink.com.au> (Anthony Baxter) Date: Fri, 19 Oct 2001 01:11:20 +1000 Subject: [Python-Dev] Re: python2.1.1 SEGV in GC on Solaris 2.7 In-Reply-To: Message from Neil Schemenauer <nas@python.ca> of "Thu, 18 Oct 2001 07:38:15 MST." <20011018073815.A22506@glacier.arctrix.com> Message-ID: <200110181511.f9IFBKu01348@mbuna.arbhome.com.au> >>> Neil Schemenauer wrote > [I'm moving the discussion here from SF, using the tracker is too > painful.] ok. please CC me on any replies. = current update - I applied the PyFrame_BlockSetup() fix for ceval.c that Martin suggested, and so far I've had 4 processes running for a couple of hours each, without any splattage. this doesn't necessarily mean it's fixed - the time between failures has varied from 5 minutes to 12 hours over the last few days. > The ob_type pointer must be getting cleared after the object has been > added to the GC lists. The PyObject_IS_GC call in _PyGC_Insert would > have segfaulted otherwise. Knowing the type of the object would be > helpful in debugging the problem. I suggest reconsidering Martin's > printf idea. You could add something like this to _PyGC_Insert: I'll have a poke at this tomorrow, and see if it's possible at all - the problem is that the boxes are part of a zope cluster that gets = a _lot_ of hits (something over 1m/day) so that's a lot of objects... I'm also going to have another go at getting purify to work on our stripped down production servers. = > Debugging this type of problem is really hard (as you already know) > because the effect of the bug is found so far away from the source. *nod* I'm aware of it. = thanks! Anthony -- = Anthony Baxter <anthony@interlink.com.au> = It's never too late to have a happy childhood. From nas@python.ca Thu Oct 18 16:19:06 2001 From: nas@python.ca (Neil Schemenauer) Date: Thu, 18 Oct 2001 08:19:06 -0700 Subject: [Python-Dev] 2.1.1 maintenance branch In-Reply-To: <200110181441.f9IEfZF02019@odiug.zope.com>; from guido@python.org on Thu, Oct 18, 2001 at 10:41:35AM -0400 References: <200110181428.QAA16670@paros.informatik.hu-berlin.de> <200110181441.f9IEfZF02019@odiug.zope.com> Message-ID: <20011018081906.A22654@glacier.arctrix.com> Guido van Rossum wrote: > If someone (you?) volunteers to act as a 2.1.2 release manager, > chances of a 2.1.2 release go way up. I'm willing to spend some time backporting patches but I pretty sure I don't have time to be the release manager. Neil From barry@zope.com Thu Oct 18 16:25:56 2001 From: barry@zope.com (Barry A. Warsaw) Date: Thu, 18 Oct 2001 11:25:56 -0400 Subject: [Python-Dev] python2.1.1 SEGV in GC on Solaris 2.7 References: <E15uASk-0004Ax-00@usw-sf-web3.sourceforge.net> <20011018073815.A22506@glacier.arctrix.com> Message-ID: <15310.62596.531353.944255@anthem.wooz.org> NS> The ob_type pointer must be getting cleared after the object NS> has been added to the GC lists. The PyObject_IS_GC call in NS> _PyGC_Insert would have segfaulted otherwise. Knowing the NS> type of the object would be helpful in debugging the problem. NS> I suggest reconsidering Martin's printf idea. You could add NS> something like this to _PyGC_Insert: Do you know about _PyObject_Dump() and _PyGC_Dump()? See also Misc/gdbinit. -Barry From thomas.heller@ion-tof.com Thu Oct 18 16:26:51 2001 From: thomas.heller@ion-tof.com (Thomas Heller) Date: Thu, 18 Oct 2001 17:26:51 +0200 Subject: [Python-Dev] Function arguments References: <011d01c157ae$eff659b0$e000a8c0@thomasnotebook> <15310.50240.333969.995440@grendel.zope.com> <15310.59190.519026.561865@beluga.mojam.com> Message-ID: <04ea01c157e9$4fc79180$e000a8c0@thomasnotebook> From: "Skip Montanaro" <skip@pobox.com> > > It's perhaps worth pointing out that dicts already have a zero-arg popitem > method. I don't think I would use Thomas's proposed popitem so frequently > that the fairly obvious > > v = d.get(k, default) > try: del d[k] > except KeyError: pass > > would be all that much trouble to type. Those are the semantics you're > proposing, right? Exactly. Thomas From loewis@informatik.hu-berlin.de Thu Oct 18 16:29:48 2001 From: loewis@informatik.hu-berlin.de (Martin von Loewis) Date: Thu, 18 Oct 2001 17:29:48 +0200 (MEST) Subject: [Python-Dev] python2.1.1 SEGV in GC on Solaris 2.7 In-Reply-To: <15310.62596.531353.944255@anthem.wooz.org> (barry@zope.com) References: <E15uASk-0004Ax-00@usw-sf-web3.sourceforge.net> <20011018073815.A22506@glacier.arctrix.com> <15310.62596.531353.944255@anthem.wooz.org> Message-ID: <200110181529.RAA19958@paros.informatik.hu-berlin.de> > Do you know about _PyObject_Dump() and _PyGC_Dump()? See also > Misc/gdbinit. In the specific problem, ob_type is NULL at the time of the crash, so _PyObject_Dump won't do anything interesting. Regards, Martin From guido@python.org Thu Oct 18 16:31:51 2001 From: guido@python.org (Guido van Rossum) Date: Thu, 18 Oct 2001 11:31:51 -0400 Subject: [Python-Dev] 2.1.1 maintenance branch In-Reply-To: Your message of "Thu, 18 Oct 2001 08:19:06 PDT." <20011018081906.A22654@glacier.arctrix.com> References: <200110181428.QAA16670@paros.informatik.hu-berlin.de> <200110181441.f9IEfZF02019@odiug.zope.com> <20011018081906.A22654@glacier.arctrix.com> Message-ID: <200110181532.f9IFVvm02497@odiug.zope.com> > I'm willing to spend some time backporting patches but I pretty sure I > don't have time to be the release manager. OK, we can do it without a release manager for a while. What kind of patches did you have in mind? --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Thu Oct 18 16:37:14 2001 From: guido@python.org (Guido van Rossum) Date: Thu, 18 Oct 2001 11:37:14 -0400 Subject: [Python-Dev] warnings in dynload_shlib.c and posixmodule.c Message-ID: <200110181537.f9IFbK204967@odiug.zope.com> When compiling the latest CVS on Linux (Mandrake 8.0) I get these warnings: ../Python/dynload_shlib.c: In function `_PyImport_GetDynLoadFunc': ../Python/dynload_shlib.c:72: warning: implicit declaration of function `dlsym' ../Python/dynload_shlib.c:88: warning: implicit declaration of function `dlopen' ../Python/dynload_shlib.c:88: warning: assignment makes pointer from integer without a cast ../Python/dynload_shlib.c:91: warning: implicit declaration of function `dlerror' ../Python/dynload_shlib.c:91: warning: passing arg 2 of `PyErr_SetString' makes pointer from integer without a cast ../Modules/posixmodule.c: In function `posix_setgroups': ../Modules/posixmodule.c:3169: warning: implicit declaration of function `setgroups' Looks like some recent checkins... --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Thu Oct 18 16:38:45 2001 From: guido@python.org (Guido van Rossum) Date: Thu, 18 Oct 2001 11:38:45 -0400 Subject: [Python-Dev] python2.1.1 SEGV in GC on Solaris 2.7 In-Reply-To: Your message of "Thu, 18 Oct 2001 17:29:48 +0200." <200110181529.RAA19958@paros.informatik.hu-berlin.de> References: <E15uASk-0004Ax-00@usw-sf-web3.sourceforge.net> <20011018073815.A22506@glacier.arctrix.com> <15310.62596.531353.944255@anthem.wooz.org> <200110181529.RAA19958@paros.informatik.hu-berlin.de> Message-ID: <200110181538.f9IFcp505270@odiug.zope.com> > > Do you know about _PyObject_Dump() and _PyGC_Dump()? See also > > Misc/gdbinit. > > In the specific problem, ob_type is NULL at the time of the crash, so > _PyObject_Dump won't do anything interesting. Plus the only way he can debug this is by waiting for a core dump to happen; I don't think those routines work in post-mortem debug sesions. --Guido van Rossum (home page: http://www.python.org/~guido/) From barry@zope.com Thu Oct 18 17:07:29 2001 From: barry@zope.com (Barry A. Warsaw) Date: Thu, 18 Oct 2001 12:07:29 -0400 Subject: [Python-Dev] python2.1.1 SEGV in GC on Solaris 2.7 References: <E15uASk-0004Ax-00@usw-sf-web3.sourceforge.net> <20011018073815.A22506@glacier.arctrix.com> <15310.62596.531353.944255@anthem.wooz.org> <200110181529.RAA19958@paros.informatik.hu-berlin.de> <200110181538.f9IFcp505270@odiug.zope.com> Message-ID: <15310.65089.228565.876912@anthem.wooz.org> >>>>> "GvR" == Guido van Rossum <guido@python.org> writes: GvR> Plus the only way he can debug this is by waiting for a core GvR> dump to happen; I don't think those routines work in GvR> post-mortem debug sesions. You can run the process under gdb or attach to it early in the startup period. Then when gdb gets the SEGV, you can often (but not always!) _PyObject_Dump() objects on the call stack. -Barry From nas@python.ca Thu Oct 18 17:19:14 2001 From: nas@python.ca (Neil Schemenauer) Date: Thu, 18 Oct 2001 09:19:14 -0700 Subject: [Python-Dev] 2.1.1 maintenance branch In-Reply-To: <200110181532.f9IFVvm02497@odiug.zope.com>; from guido@python.org on Thu, Oct 18, 2001 at 11:31:51AM -0400 References: <200110181428.QAA16670@paros.informatik.hu-berlin.de> <200110181441.f9IEfZF02019@odiug.zope.com> <20011018081906.A22654@glacier.arctrix.com> <200110181532.f9IFVvm02497@odiug.zope.com> Message-ID: <20011018091913.A22846@glacier.arctrix.com> Guido van Rossum wrote: > OK, we can do it without a release manager for a while. What kind of > patches did you have in mind? So far: Thomas's ceval fix, profile.py, abstract_isinstance. I'm sure there's lots more if I start looking through the logs. Is it worthing fixing the alignment of the GC header? Neil From fredrik@pythonware.com Thu Oct 18 17:35:52 2001 From: fredrik@pythonware.com (Fredrik Lundh) Date: Thu, 18 Oct 2001 18:35:52 +0200 Subject: [Python-Dev] 2.2b1 release plan? Message-ID: <003701c157f2$f6ab05f0$ced241d5@hagrid> the PEP says October 10. I was planning to do some SRE work tonight, but I'm not sure I'll be able to squeeze it in. when is 2.2b1 going out? will I have time to work on this over the weekend? </F> From guido@python.org Thu Oct 18 17:31:20 2001 From: guido@python.org (Guido van Rossum) Date: Thu, 18 Oct 2001 12:31:20 -0400 Subject: [Python-Dev] 2.1.1 maintenance branch In-Reply-To: Your message of "Thu, 18 Oct 2001 09:19:14 PDT." <20011018091913.A22846@glacier.arctrix.com> References: <200110181428.QAA16670@paros.informatik.hu-berlin.de> <200110181441.f9IEfZF02019@odiug.zope.com> <20011018081906.A22654@glacier.arctrix.com> <200110181532.f9IFVvm02497@odiug.zope.com> <20011018091913.A22846@glacier.arctrix.com> Message-ID: <200110181631.f9IGVLR11199@odiug.zope.com> > > OK, we can do it without a release manager for a while. What kind of > > patches did you have in mind? > > So far: Thomas's ceval fix, profile.py, abstract_isinstance. I'm sure > there's lots more if I start looking through the logs. Is it worthing > fixing the alignment of the GC header? I think so. I'n curious about abstract_isinstance -- is that what you checked in last night? I thought that was for 2.2 but I may have misunderstood the issue (I have to admit I have no idea what abstract_isinstance does any more :-( ). --Guido van Rossum (home page: http://www.python.org/~guido/) From tim@zope.com Thu Oct 18 17:35:57 2001 From: tim@zope.com (Tim Peters) Date: Thu, 18 Oct 2001 12:35:57 -0400 Subject: [Python-Dev] 2.1.1 maintenance branch In-Reply-To: <20011018091913.A22846@glacier.arctrix.com> Message-ID: <BIEJKCLHCIOIHAGOKOLHKENPCIAA.tim@zope.com> [Neil Schemenauer] > Is it worthing fixing the alignment of the GC header? I wouldn't -- there was nothing in the core in 2.1.1 that both participated in gc and required double alignment, and no reports of extension modules that care. From guido@python.org Thu Oct 18 17:35:22 2001 From: guido@python.org (Guido van Rossum) Date: Thu, 18 Oct 2001 12:35:22 -0400 Subject: [Python-Dev] 2.2b1 release plan? In-Reply-To: Your message of "Thu, 18 Oct 2001 18:35:52 +0200." <003701c157f2$f6ab05f0$ced241d5@hagrid> References: <003701c157f2$f6ab05f0$ced241d5@hagrid> Message-ID: <200110181635.f9IGZMM11249@odiug.zope.com> > the PEP says October 10. Sorry, we slipped by about the same amount as 2.2a4. It goes out tomorrow. > I was planning to do some SRE work tonight, but > I'm not sure I'll be able to squeeze it in. when is > 2.2b1 going out? will I have time to work on this > over the weekend? Argh! Please squeeze it in tonight. Fortunately your tonight is earlier than ours. :) --Guido van Rossum (home page: http://www.python.org/~guido/) From tim@zope.com Thu Oct 18 17:38:02 2001 From: tim@zope.com (Tim Peters) Date: Thu, 18 Oct 2001 12:38:02 -0400 Subject: [Python-Dev] 2.2b1 release plan? In-Reply-To: <003701c157f2$f6ab05f0$ced241d5@hagrid> Message-ID: <BIEJKCLHCIOIHAGOKOLHCEOACIAA.tim@zope.com> [/F] > I was planning to do some SRE work tonight, but > I'm not sure I'll be able to squeeze it in. when is > 2.2b1 going out? Tomorrow (Friday). > will I have time to work on this over the weekend? I hope so <wink>. From guido@python.org Thu Oct 18 17:55:44 2001 From: guido@python.org (Guido van Rossum) Date: Thu, 18 Oct 2001 12:55:44 -0400 Subject: [Python-Dev] 2.1.1 maintenance branch In-Reply-To: Your message of "Thu, 18 Oct 2001 12:35:57 EDT." <BIEJKCLHCIOIHAGOKOLHKENPCIAA.tim@zope.com> References: <BIEJKCLHCIOIHAGOKOLHKENPCIAA.tim@zope.com> Message-ID: <200110181655.f9IGti111355@odiug.zope.com> > [Neil Schemenauer] > > Is it worthing fixing the alignment of the GC header? [Tim] > I wouldn't -- there was nothing in the core in 2.1.1 that both participated > in gc and required double alignment, and no reports of extension modules > that care. You're right. I forgot about this. Let's not do this one. --Guido van Rossum (home page: http://www.python.org/~guido/) From barry@zope.com Thu Oct 18 18:58:40 2001 From: barry@zope.com (Barry A. Warsaw) Date: Thu, 18 Oct 2001 13:58:40 -0400 Subject: [Python-Dev] 2.2b1 release plan? References: <003701c157f2$f6ab05f0$ced241d5@hagrid> Message-ID: <15311.6224.239595.715872@anthem.wooz.org> >>>>> "FL" == Fredrik Lundh <fredrik@pythonware.com> writes: FL> the PEP says October 10. The PEP has been updated. -Barry From nas@python.ca Thu Oct 18 20:53:22 2001 From: nas@python.ca (Neil Schemenauer) Date: Thu, 18 Oct 2001 12:53:22 -0700 Subject: [Python-Dev] 2.1.1 maintenance branch In-Reply-To: <200110181631.f9IGVLR11199@odiug.zope.com>; from guido@python.org on Thu, Oct 18, 2001 at 12:31:20PM -0400 References: <200110181428.QAA16670@paros.informatik.hu-berlin.de> <200110181441.f9IEfZF02019@odiug.zope.com> <20011018081906.A22654@glacier.arctrix.com> <200110181532.f9IFVvm02497@odiug.zope.com> <20011018091913.A22846@glacier.arctrix.com> <200110181631.f9IGVLR11199@odiug.zope.com> Message-ID: <20011018125322.A23424@glacier.arctrix.com> Guido van Rossum wrote: > I'n curious about abstract_isinstance -- is that what you checked in > last night? Yes. > I thought that was for 2.2 but I may have misunderstood > the issue (I have to admit I have no idea what abstract_isinstance > does any more :-( ). No, abstract_isinstance has been broken for a long time, maybe forever. It is supposed to work for class-like objects like ExtensionClass instances as well as for standard classes and types. One problem was that if you gave it a normal instance as the first argument but an ExtensionClass as the second argument it raised a type error. It's annoying to us at the MEMS Exchange because we use ExtensionClasses and isinstance all the time (Greg likes type checking). We ended up rewriting isinstance() and issubclass() in Python. The average Python user doesn't care. Neil From guido@python.org Thu Oct 18 20:58:20 2001 From: guido@python.org (Guido van Rossum) Date: Thu, 18 Oct 2001 15:58:20 -0400 Subject: [Python-Dev] 2.1.1 maintenance branch In-Reply-To: Your message of "Thu, 18 Oct 2001 12:53:22 PDT." <20011018125322.A23424@glacier.arctrix.com> References: <200110181428.QAA16670@paros.informatik.hu-berlin.de> <200110181441.f9IEfZF02019@odiug.zope.com> <20011018081906.A22654@glacier.arctrix.com> <200110181532.f9IFVvm02497@odiug.zope.com> <20011018091913.A22846@glacier.arctrix.com> <200110181631.f9IGVLR11199@odiug.zope.com> <20011018125322.A23424@glacier.arctrix.com> Message-ID: <200110181958.f9IJwKA29065@odiug.zope.com> > > I'n curious about abstract_isinstance -- is that what you checked in > > last night? > > Yes. > > > I thought that was for 2.2 but I may have misunderstood > > the issue (I have to admit I have no idea what abstract_isinstance > > does any more :-( ). > > No, abstract_isinstance has been broken for a long time, maybe forever. > It is supposed to work for class-like objects like ExtensionClass > instances as well as for standard classes and types. One problem was > that if you gave it a normal instance as the first argument but an > ExtensionClass as the second argument it raised a type error. It's > annoying to us at the MEMS Exchange because we use ExtensionClasses and > isinstance all the time (Greg likes type checking). We ended up > rewriting isinstance() and issubclass() in Python. The average Python > user doesn't care. Then I'm not sure if this should be fixed in 2.1.2. It's almost a new feature that it doesn't bomb out. :-( --Guido van Rossum (home page: http://www.python.org/~guido/) From thomas.heller@ion-tof.com Thu Oct 18 21:11:07 2001 From: thomas.heller@ion-tof.com (Thomas Heller) Date: Thu, 18 Oct 2001 22:11:07 +0200 Subject: [Python-Dev] 2.1.1 maintenance branch References: <200110181428.QAA16670@paros.informatik.hu-berlin.de> <200110181441.f9IEfZF02019@odiug.zope.com> <20011018081906.A22654@glacier.arctrix.com> <200110181532.f9IFVvm02497@odiug.zope.com> Message-ID: <070b01c15811$05e3d470$e000a8c0@thomasnotebook> > > OK, we can do it without a release manager for a while. What kind of > patches did you have in mind? Another bug which should be repaired in a 2.1.2 release is that distutils version number is invalid, see [ 417796 ] illegal version 1.0.2pre (priority 5) http://sf.net/tracker/?func=detail&aid=417796&group_id=5470&atid=105470 Thomas From fdrake@acm.org Thu Oct 18 21:10:34 2001 From: fdrake@acm.org (Fred L. Drake) Date: Thu, 18 Oct 2001 16:10:34 -0400 (EDT) Subject: [Python-Dev] [development doc updates] Message-ID: <20011018201034.72D8E28697@cj42289-a.reston1.va.home.com> The development version of the documentation has been updated: http://python.sourceforge.net/devel-docs/ Documentation as packaged with Python 2.2 beta 1. From skip@pobox.com (Skip Montanaro) Thu Oct 18 21:17:15 2001 From: skip@pobox.com (Skip Montanaro) (Skip Montanaro) Date: Thu, 18 Oct 2001 15:17:15 -0500 Subject: [Python-Dev] 2.1.1 maintenance branch In-Reply-To: <20011018125322.A23424@glacier.arctrix.com> References: <200110181428.QAA16670@paros.informatik.hu-berlin.de> <200110181441.f9IEfZF02019@odiug.zope.com> <20011018081906.A22654@glacier.arctrix.com> <200110181532.f9IFVvm02497@odiug.zope.com> <20011018091913.A22846@glacier.arctrix.com> <200110181631.f9IGVLR11199@odiug.zope.com> <20011018125322.A23424@glacier.arctrix.com> Message-ID: <15311.14539.305244.20323@beluga.mojam.com> Neil> No, abstract_isinstance has been broken for a long time, maybe Neil> forever. It is supposed to work for class-like objects like Neil> ExtensionClass instances as well as for standard classes and Neil> types. One problem was that if you gave it a normal instance as Neil> the first argument but an ExtensionClass as the second argument it Neil> raised a type error. It's annoying to us at the MEMS Exchange Neil> because we use ExtensionClasses and isinstance all the time ... Getting a little off the subject, but a good first test of the new 2.2 type/class stuff is that it only took James Henstridge a couple days to convert the PyGtk2 stuff from ExtensionClass to pure 2.2 code. I've experienced no problems related to that change. Skip From fdrake@acm.org Thu Oct 18 21:11:31 2001 From: fdrake@acm.org (Fred L. Drake, Jr.) Date: Thu, 18 Oct 2001 16:11:31 -0400 Subject: [Python-Dev] 2.1.1 maintenance branch In-Reply-To: <070b01c15811$05e3d470$e000a8c0@thomasnotebook> References: <200110181428.QAA16670@paros.informatik.hu-berlin.de> <200110181441.f9IEfZF02019@odiug.zope.com> <20011018081906.A22654@glacier.arctrix.com> <200110181532.f9IFVvm02497@odiug.zope.com> <070b01c15811$05e3d470$e000a8c0@thomasnotebook> Message-ID: <15311.14195.862624.167879@grendel.zope.com> Guido writes: > OK, we can do it without a release manager for a while. What kind of > patches did you have in mind? The diff between Objects/weakrefobject.c 1.1 and 1.3 should be applied to Modules/_weakref.c for 2.1.2, since that could cause core dumps. -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> PythonLabs at Zope Corporation From fredrik@pythonware.com Thu Oct 18 22:06:53 2001 From: fredrik@pythonware.com (Fredrik Lundh) Date: Thu, 18 Oct 2001 23:06:53 +0200 Subject: [Python-Dev] Re: [Python-checkins] CVS: python/dist/src/Lib/test test_os.py,1.6,1.7 References: <E15uJs3-0002JX-00@usw-pr-cvs1.sourceforge.net> Message-ID: <012d01c15818$d62b0ed0$ced241d5@hagrid> guido wrote: > But they also have attributes like st_mtime or tm_year. umm. do we really need the "st_" and "tm_" prefixes? </F> From fdrake@acm.org Thu Oct 18 22:00:20 2001 From: fdrake@acm.org (Fred L. Drake, Jr.) Date: Thu, 18 Oct 2001 17:00:20 -0400 Subject: [Python-Dev] Python 2.2 beta 1 documentation Message-ID: <15311.17124.615391.305548@grendel.zope.com> The Python 2.2 beta 1 documentation is now available in the usual places. The HTML packages are there as always, and one new package: a version for the iSilo document reader for PalmOS devices (see www.iSilo.com). It has problems still, but as long as you navigate by hyperlinks it should work just fine. ;-) If people are interested in this format I can work on solving the problems with it, so please let me know if you think you'll be interested in actually using docs in this format. -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> PythonLabs at Zope Corporation From guido@python.org Thu Oct 18 22:08:50 2001 From: guido@python.org (Guido van Rossum) Date: Thu, 18 Oct 2001 17:08:50 -0400 Subject: [Python-Dev] Re: [Python-checkins] CVS: python/dist/src/Lib/test test_os.py,1.6,1.7 In-Reply-To: Your message of "Thu, 18 Oct 2001 23:06:53 +0200." <012d01c15818$d62b0ed0$ced241d5@hagrid> References: <E15uJs3-0002JX-00@usw-pr-cvs1.sourceforge.net> <012d01c15818$d62b0ed0$ced241d5@hagrid> Message-ID: <200110182108.f9IL8pg03390@odiug.zope.com> > > But they also have attributes like st_mtime or tm_year. > > umm. do we really need the "st_" and "tm_" prefixes? I think so. They make it easier to switch between C and Python code, documentation, brains etc. --Guido van Rossum (home page: http://www.python.org/~guido/) From fredrik@pythonware.com Thu Oct 18 22:29:52 2001 From: fredrik@pythonware.com (Fredrik Lundh) Date: Thu, 18 Oct 2001 23:29:52 +0200 Subject: [Python-Dev] Re: [Python-checkins] CVS: python/dist/src/Lib/test test_os.py,1.6,1.7 References: <E15uJs3-0002JX-00@usw-pr-cvs1.sourceforge.net> <012d01c15818$d62b0ed0$ced241d5@hagrid> <200110182108.f9IL8pg03390@odiug.zope.com> Message-ID: <001f01c1581c$0c2a6c80$ced241d5@hagrid> Guido van Rossum wrote: > > umm. do we really need the "st_" and "tm_" prefixes? > > I think so. They make it easier to switch between C and Python > code, documentation, brains etc. does that imply that tm.tm_year != tm[0]? and should really all those of us who wouldn't dream of writing time or stat code in C have to suffer? -0.5 from me. </F> From gward@python.net Fri Oct 19 00:00:06 2001 From: gward@python.net (Greg Ward) Date: Thu, 18 Oct 2001 19:00:06 -0400 Subject: [Python-Dev] CVS question: checking in old doc update Message-ID: <20011018190006.A6046@gerg.ca> How embarassing. I've just discovered that I added a couple pages of text to one of the Distutils docs a year ago, and then never checked it in. Since then, various people have made various changes, and simply "cvs up"'ing produced a horrible mess. For posterity, I'd like to check in my changes relative to the last revision I worked on. That is, sometime in October 2000, I made revision 1.26 to Doc/dist/dist.tex, and on October 29th (according to my own backups), I added a bunch of text and made some other changes. So I *think* I'd like to make a "mini-branch" for this one file, where I check in my year-old changes relative to revision 1.26. I tried this: cvs ci -r1.26 dist.tex but CVS said cvs server: Up-to-date check failed for `dist.tex' So I said, "Fine, I'll be explicit about working relative to rev 1.26": mv dist.tex dist.tex-hacked cvs up -r 1.26 dist.tex mv dist.tex-hacked dist.tex cvs ci dist.tex And this time CVS said: cvs server: sticky tag `1.26' for file `dist.tex' is not a branch Sigh. Do I have to explicitly create a branch for this, as in cvs tag -r 1.26 -b last-gpw-revision ? If that's the only thing to do, is it OK for me to do this? It's only one file. Is that a good name for the branch? Greg -- Greg Ward - just another /P(erl|ython)/ hacker gward@python.net http://starship.python.net/~gward/ Reality is for people who can't handle science fiction. From guido@python.org Fri Oct 19 00:27:24 2001 From: guido@python.org (Guido van Rossum) Date: Thu, 18 Oct 2001 19:27:24 -0400 Subject: [Python-Dev] Re: [Python-checkins] CVS: python/dist/src/Lib/test test_os.py,1.6,1.7 In-Reply-To: Your message of "Thu, 18 Oct 2001 23:29:52 +0200." <001f01c1581c$0c2a6c80$ced241d5@hagrid> References: <E15uJs3-0002JX-00@usw-pr-cvs1.sourceforge.net> <012d01c15818$d62b0ed0$ced241d5@hagrid> <200110182108.f9IL8pg03390@odiug.zope.com> <001f01c1581c$0c2a6c80$ced241d5@hagrid> Message-ID: <200110182327.TAA06329@cj20424-a.reston1.va.home.com> > does that imply that tm.tm_year != tm[0]? Fortunately not. > and should really all those of us who wouldn't dream of > writing time or stat code in C have to suffer? -0.5 from > me. Call me old-fashioned, but I *like* struct fields with unique prefixes, in Python as well as in C. It's a hint about what kind of data structure we're looking at. And it greps nicely. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Fri Oct 19 00:32:44 2001 From: guido@python.org (Guido van Rossum) Date: Thu, 18 Oct 2001 19:32:44 -0400 Subject: [Python-Dev] CVS question: checking in old doc update In-Reply-To: Your message of "Thu, 18 Oct 2001 19:00:06 EDT." <20011018190006.A6046@gerg.ca> References: <20011018190006.A6046@gerg.ca> Message-ID: <200110182332.TAA06390@cj20424-a.reston1.va.home.com> > Sigh. Do I have to explicitly create a branch for this, as in > > cvs tag -r 1.26 -b last-gpw-revision > > ? Yes. > If that's the only thing to do, is it OK for me to do this? It's > only one file. Is that a good name for the branch? Sure. --Guido van Rossum (home page: http://www.python.org/~guido/) From DavidA@ActiveState.com Fri Oct 19 01:19:22 2001 From: DavidA@ActiveState.com (David Ascher) Date: Thu, 18 Oct 2001 17:19:22 -0700 Subject: [Python-Dev] Re: [Python-checkins] CVS: python/dist/src/Lib/test test_os.py,1.6,1. References: <E15uJs3-0002JX-00@usw-pr-cvs1.sourceforge.net> <012d01c15818$d62b0ed0$ced241d5@hagrid> <200110182108.f9IL8pg03390@odiug.zope.com> <001f01c1581c$0c2a6c80$ced241d5@hagrid> <200110182327.TAA06329@cj20424-a.reston1.va.home.com> Message-ID: <3BCF718A.1050501@ActiveState.com> Guido van Rossum wrote: > Call me old-fashioned, but I *like* struct fields with unique > prefixes, in Python as well as in C. It's a hint about what kind of > data structure we're looking at. And it greps nicely. Ok. "Old fashioned!" --david PS: FWIW, I'm w/ /F on this one. It never made sense to me why I had to type: traceback.tb_frame.f_code.co_consts instead of traceback.frame.code.constants Not a huge deal, though, in the scheme of things. =) From paulp@ActiveState.com Fri Oct 19 01:48:46 2001 From: paulp@ActiveState.com (Paul Prescod) Date: Thu, 18 Oct 2001 17:48:46 -0700 Subject: [Python-Dev] Re: [Python-checkins] CVS: python/dist/src/Lib/test test_os.py,1.6,1. References: <E15uJs3-0002JX-00@usw-pr-cvs1.sourceforge.net> <012d01c15818$d62b0ed0$ced241d5@hagrid> <200110182108.f9IL8pg03390@odiug.zope.com> <001f01c1581c$0c2a6c80$ced241d5@hagrid> <200110182327.TAA06329@cj20424-a.reston1.va.home.com> <3BCF718A.1050501@ActiveState.com> Message-ID: <3BCF786E.A901374A@ActiveState.com> David Ascher wrote: > >... > PS: FWIW, I'm w/ /F on this one. It never made sense to me why I had to > type: > > traceback.tb_frame.f_code.co_consts > > instead of > > traceback.frame.code.constants I never liked that either. Plus Python doesn't have a real distinction between "structs" and "objects" so it is just one more thing to remember: "does this thing use the obsolete C struct convention or the modern Python object convention?" Paul Prescod From fdrake@acm.org Fri Oct 19 03:40:27 2001 From: fdrake@acm.org (Fred L. Drake, Jr.) Date: Thu, 18 Oct 2001 22:40:27 -0400 Subject: [Python-Dev] CVS question: checking in old doc update In-Reply-To: <20011018190006.A6046@gerg.ca> References: <20011018190006.A6046@gerg.ca> <200110182332.TAA06390@cj20424-a.reston1.va.home.com> Message-ID: <15311.37531.900530.578011@grendel.zope.com> Greg Ward writes: > ? If that's the only thing to do, is it OK for me to do this? It's > only one file. Is that a good name for the branch? Please don't! Ignore what Guido said -- the docs are mine. ;-) Submit the difference between what you have and what you started from as a context diff on SourceForge; assign it to me. -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> PythonLabs at Zope Corporation From greg@cosc.canterbury.ac.nz Fri Oct 19 04:52:38 2001 From: greg@cosc.canterbury.ac.nz (Greg Ewing) Date: Fri, 19 Oct 2001 16:52:38 +1300 (NZDT) Subject: [Python-Dev] Re: [Python-checkins] CVS: python/dist/src/Lib/test test_os.py,1.6,1.7 In-Reply-To: <200110182327.TAA06329@cj20424-a.reston1.va.home.com> Message-ID: <200110190352.QAA07859@s454.cosc.canterbury.ac.nz> Guido: > Call me old-fashioned, but I *like* struct fields with unique > prefixes, in Python as well as in C. It's a hint about what kind of > data structure we're looking at. The name of whatever is referring to the struct should be doing that. Anyway, if it's such a good thing, why don't you use it in all the Python code you write? Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From Mark.Favas@csiro.au Fri Oct 19 04:54:41 2001 From: Mark.Favas@csiro.au (Mark.Favas@csiro.au) Date: Fri, 19 Oct 2001 11:54:41 +0800 Subject: [Python-Dev] Has socketmodule compilation broken for anyone else lately? Message-ID: <116D27C8E12BD411B3AB00B0D022B0B87C46D2@yate.wa.CSIRO.AU> For the last couple of days, socketmodule hasn't compiled for me on Tru64 Unix. If I'm the only one seeing this, I'll file a bug report on sf... building '_socket' extension cc: Error: Modules/socketmodule.c, line 2820: An unexpected newline character is present in a string literal. (nlstring) "RAND_egd(path) -> bytes ^ cc: Warning: Modules/socketmodule.c, line 2821: Invalid token discarded. (invaltoken) \n\ ^ cc: Warning: Modules/socketmodule.c, line 2822: Invalid token discarded. (invaltoken) Queries the entropy gather daemon (EGD) on socket path. Returns number\n\ -----------------------------------------------------------------------^ cc: Warning: Modules/socketmodule.c, line 2823: Invalid token discarded. (invaltoken) of bytes read. Raises socket.sslerror if connection to EGD fails or\n\ --------------------------------------------------------------------^ cc: Error: Modules/socketmodule.c, line 2824: An unexpected newline character is present in a string literal. (nlstring) if it does provide enough data to seed PRNG."; --------------------------------------------^ cc: Error: Modules/socketmodule.c, line 3037: In this statement, "PySocket_methods" is not declared. (undeclared) m = Py_InitModule3("_socket", PySocket_methods, module_doc); ------------^ /bin/cc -O -Olimit 1500 -DUSE_SSL=1 -I/usr/local/ssl/include -I. -I/home/erebus2/groucho1/mark/src/python/CVS/python/dist/src/./Include -I/usr/local/include -IInclude/ -c /home/erebus2/groucho1/mark/src/python/CVS/python/dist/src/Modules/socketmod ule.c -o build/temp.osf1-V4.0-alpha-2.2/socketmodule.o WARNING: building of extension "_socket" failed: command '/bin/cc' failed with exit status 1 From anthony@interlink.com.au Fri Oct 19 05:31:37 2001 From: anthony@interlink.com.au (Anthony Baxter) Date: Fri, 19 Oct 2001 14:31:37 +1000 Subject: [Python-Dev] Goals for patch selection for 2.1.2 Message-ID: <200110190431.f9J4VbL04554@mbuna.arbhome.com.au> Hi there, I'm happy enough to be the release manager for 2.1.2 - I've a reasonably high amount of CVS experience, so that shouldn't be a problem. Here's my current thoughts for selecting patches for a 2.1.2 release. Feedback would be appreciated. I plan to post this to python-list and ask people for suggestions for "must have" fixes. This could possibly be added to PEP 006 once it's nailed down. Code that adds new packages new modules new methods is out. (duh) Anything that introduces a large chunk of code should be out. Anyone who's subclassing a standard library class shouldn't get screwed over - even if they're playing with the internals of the classes. C code changes should be _very_ _very_ conservative. Bug fixes that fix problems that people are having are good - anything else should be regarded with a hell of a lot of suspicion. And the "don't introduce large chunks of new code" test would obviously apply here. Things that shouldn't be considered: Fixes that "make something work better" Fixes that "remove a nasty workaround" In general, I'd like the starting point for any patch to be considered as "guilty until proven necessary". An example (from a discussion with Guido) is the recent profiler fixup: We made two sets of changes to the profiler. One fixed profile.py to properly interpret exception events. This one should go in (I specifically tested it with Python 1.5.2 and 2.1). The other changed the C code that generates profiling events to suppress exception events, and to generate return events even when a frame is exited with an exception. That one should not go in. Thanks, Anthony -- Anthony Baxter <anthony@interlink.com.au> It's never too late to have a happy childhood. From barry@zope.com Fri Oct 19 05:39:49 2001 From: barry@zope.com (Barry A. Warsaw) Date: Fri, 19 Oct 2001 00:39:49 -0400 Subject: [Python-Dev] Has socketmodule compilation broken for anyone else lately? References: <116D27C8E12BD411B3AB00B0D022B0B87C46D2@yate.wa.CSIRO.AU> Message-ID: <15311.44693.311402.326528@anthem.wooz.org> >>>>> "MF" == Mark Favas <Mark.Favas@csiro.au> writes: MF> For the last couple of days, socketmodule hasn't compiled for MF> me on Tru64 Unix. If I'm the only one seeing this, I'll file a MF> bug report on sf... I just did a fresh cvs update, "make test" on Linux and everything works for me. I'm about to create the 2.2b1 release branch, so unless the patch is in my inbox when I wake up about 8 hours from now, it probably won't make it into 2.2b1. -Barry From Anthony Baxter <anthony@interlink.com.au> Fri Oct 19 05:52:33 2001 From: Anthony Baxter <anthony@interlink.com.au> (Anthony Baxter) Date: Fri, 19 Oct 2001 14:52:33 +1000 Subject: [Python-Dev] python2.1.1 SEGV in GC on Solaris 2.7 In-Reply-To: Message from Martin von Loewis <loewis@informatik.hu-berlin.de> of "Thu, 18 Oct 2001 17:29:48 +0200." <200110181529.RAA19958@paros.informatik.hu-berlin.de> Message-ID: <200110190452.f9J4qXX04666@mbuna.arbhome.com.au> Ok, after more than 16 hours of running, I can say that I'm pretty confident the crash-in-realloc is gone. However, I'm still seeing crashes (although much much less frequent) inside the GC. Same bat time, same bat channel, same bat line-of-code. I'm thinking I'll have to try the printf approach :-/ I might wait until next week, because our local directors will be back on line then, and it will be easier to knock servers in and out of service. Anthony From barry@zope.com Fri Oct 19 06:51:06 2001 From: barry@zope.com (Barry A. Warsaw) Date: Fri, 19 Oct 2001 01:51:06 -0400 Subject: [Python-Dev] Python 2.2b1 - release branch created Message-ID: <15311.48970.398745.60515@anthem.wooz.org> I've created the Python 2.2b1 release branch. The CVS tag is `r22b1-branch'. There's also a tag `r22b1-fork' -- that's the point on the trunk where the branch forks off. So far the only change in the branch is the version number. As usual, no one except Guido and myself (or our authorized bots) should make checkins to the branch. Checkins to the trunk are still okay, but generally will /not/ be merged into the release branch unless you notify us (a comment in the checkin message would suffice). I'll be cutting the release tomorrow morning local time (Friday October 19). After the release is done, the branch will be merged back into the trunk. Please note too that with the release of 2.2 beta 1, there is now a feature freeze on the trunk until after the 2.2 final release is made. According to PEP 251, this is currently scheduled for 19-Dec-2001. Those of you with commit privileges, please respect the feature freeze! You can use SourceForge to archive any feature requests or feature patches for 2.3. If you're unsure whether a change you want to make is a feature or a bug fix, contact us first. Enjoy, -Barry From loewis@informatik.hu-berlin.de Fri Oct 19 07:17:30 2001 From: loewis@informatik.hu-berlin.de (Martin von Loewis) Date: Fri, 19 Oct 2001 08:17:30 +0200 (MEST) Subject: [Python-Dev] Goals for patch selection for 2.1.2 Message-ID: <200110190617.IAA12488@pandora.informatik.hu-berlin.de> Hi Anthony, Your write-up of management principles for 2.1.2 sounds very reasonable, I think all these principles should be enforced. One open issue would be to come up with a release schedule. Should you restrict yourself to patches that also appear in 2.2 (perhaps in a back-ported fashion), or will you give a deadline for new patches to come in? If the former, I think releasing 2.1.2 around the time that 2.2 is released (perhaps earlier) would be desirable. Thanks for volunteering, Martin From Mark.Favas@csiro.au Fri Oct 19 08:40:33 2001 From: Mark.Favas@csiro.au (Mark.Favas@csiro.au) Date: Fri, 19 Oct 2001 15:40:33 +0800 Subject: [Python-Dev] Has socketmodule compilation broken for anyone e lse lately? Message-ID: <116D27C8E12BD411B3AB00B0D022B0B87C46D4@yate.wa.CSIRO.AU> Bug now logged on SourceForge... Not the first time that checkins to socketmodule have caused compilation errors (but not on Linux <wink>). Mark >>>>> "MF" == Mark Favas <Mark.Favas@csiro.au> writes: MF> For the last couple of days, socketmodule hasn't compiled for MF> me on Tru64 Unix. If I'm the only one seeing this, I'll file a MF> bug report on sf... I just did a fresh cvs update, "make test" on Linux and everything works for me. I'm about to create the 2.2b1 release branch, so unless the patch is in my inbox when I wake up about 8 hours from now, it probably won't make it into 2.2b1. -Barry From mwh@python.net Fri Oct 19 09:23:52 2001 From: mwh@python.net (Michael Hudson) Date: 19 Oct 2001 04:23:52 -0400 Subject: [Python-Dev] Re: [Python-checkins] CVS: python/dist/src/Lib __phello__.foo.py,NONE,1.1 In-Reply-To: Guido van Rossum's message of "Thu, 18 Oct 2001 11:49:39 -0700" References: <E15uIEd-0002gI-00@usw-pr-cvs1.sourceforge.net> Message-ID: <2mr8s0rqav.fsf@starship.python.net> Guido van Rossum <gvanrossum@users.sourceforge.net> writes: > Update of /cvsroot/python/python/dist/src/Lib > In directory usw-pr-cvs1:/tmp/cvs-serv9628/Lib > > Added Files: > __phello__.foo.py $ cvs up && ./configure --prefix=$HOME && make install ... $ ls ~/lib/python2.2/__phello__.foo.py /home/crew/mwh/lib/python2.2/__phello__.foo.py ...? Shouldn't it at least be in /home/crew/mwh/lib/python2.2/test ? Cheers, M. From guido@python.org Fri Oct 19 13:29:07 2001 From: guido@python.org (Guido van Rossum) Date: Fri, 19 Oct 2001 08:29:07 -0400 Subject: [Python-Dev] Has socketmodule compilation broken for anyone else lately? In-Reply-To: Your message of "Fri, 19 Oct 2001 11:54:41 +0800." <116D27C8E12BD411B3AB00B0D022B0B87C46D2@yate.wa.CSIRO.AU> References: <116D27C8E12BD411B3AB00B0D022B0B87C46D2@yate.wa.CSIRO.AU> Message-ID: <200110191229.IAA26121@cj20424-a.reston1.va.home.com> > For the last couple of days, socketmodule hasn't compiled for me on Tru64 > Unix. If I'm the only one seeing this, I'll file a bug report on sf... It's probably a missing \n\ in a string literal. I've fixed this on the release branch now. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Fri Oct 19 13:30:48 2001 From: guido@python.org (Guido van Rossum) Date: Fri, 19 Oct 2001 08:30:48 -0400 Subject: [Python-Dev] Has socketmodule compilation broken for anyone e lse lately? In-Reply-To: Your message of "Fri, 19 Oct 2001 15:40:33 +0800." <116D27C8E12BD411B3AB00B0D022B0B87C46D4@yate.wa.CSIRO.AU> References: <116D27C8E12BD411B3AB00B0D022B0B87C46D4@yate.wa.CSIRO.AU> Message-ID: <200110191230.IAA26163@cj20424-a.reston1.va.home.com> > Bug now logged on SourceForge... Not the first time that checkins to > socketmodule have caused compilation errors (but not on Linux <wink>). We should really figure out which gcc command line option makes it complain about bare newlines in string literals. Usually, the Windows build catches these, but this time it was inside an #ifdef SSL... --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Fri Oct 19 13:31:55 2001 From: guido@python.org (Guido van Rossum) Date: Fri, 19 Oct 2001 08:31:55 -0400 Subject: [Python-Dev] Re: [Python-checkins] CVS: python/dist/src/Lib __phello__.foo.py,NONE,1.1 In-Reply-To: Your message of "19 Oct 2001 04:23:52 EDT." <2mr8s0rqav.fsf@starship.python.net> References: <E15uIEd-0002gI-00@usw-pr-cvs1.sourceforge.net> <2mr8s0rqav.fsf@starship.python.net> Message-ID: <200110191231.IAA26183@cj20424-a.reston1.va.home.com> > Guido van Rossum <gvanrossum@users.sourceforge.net> writes: > > > Update of /cvsroot/python/python/dist/src/Lib > > In directory usw-pr-cvs1:/tmp/cvs-serv9628/Lib > > > > Added Files: > > __phello__.foo.py > > $ cvs up && ./configure --prefix=$HOME && make install > ... > $ ls ~/lib/python2.2/__phello__.foo.py > /home/crew/mwh/lib/python2.2/__phello__.foo.py > > ...? Shouldn't it at least be in > > /home/crew/mwh/lib/python2.2/test > > ? No, because that's inside a package (depending on how you run the test suite) and the test requires a file on sys.path. --Guido van Rossum (home page: http://www.python.org/~guido/) From thomas.heller@ion-tof.com Fri Oct 19 13:49:40 2001 From: thomas.heller@ion-tof.com (Thomas Heller) Date: Fri, 19 Oct 2001 14:49:40 +0200 Subject: [Python-Dev] Bug 216405 Message-ID: <0ad801c1589c$8592fdf0$e000a8c0@thomasnotebook> I know most of you are busy with the release, but may I beg again that SF bug #216405 be finally fixed in Python 2.2? I can apply the patch myself, if nobody has time, but I need an OK from Guido ;-) The bug is currently marked closed, resolution is set to 'later'. http://sourceforge.net/tracker/index.php?func=detail&aid=216405&group_id=5470&atid=105470 I know that the buffer stuff is somewhat (broken|fragile|dangerous, pick your favorite here), but this does at least fix a small point. Here is the complete patch against the current CVS sources, I've made a build with this patch applied, and the whole test suite runs the same way as it does without this patch (can only test on windows): 147 tests OK. 34 tests skipped: test_al test_bsddb test_cd test_cl test_commands test_crypt test_dbm test_dl test_fcntl test_fork1 test_gdbm test_gl test_grp test_gzip test_imgfile test_largefile test_linuxaudiodev test_mhlib test_minidom test_nis test_openpty test_poll test_pty test_pwd test_pyexpat test_sax test_signal test_socket_ssl test_socketserver test_sunaudiodev test_sundry test_timing test_zipfile test_zlib 8 skips unexpected on win32: test_zipfile test_bsddb test_sax test_sundry test_minidom test_zlib test_pyexpat test_gzip Thomas Index: bufferobject.c =================================================================== RCS file: /cvsroot/python/python/dist/src/Objects/bufferobject.c,v retrieving revision 2.15 diff -c -r2.15 bufferobject.c *** bufferobject.c 2001/08/24 18:34:26 2.15 --- bufferobject.c 2001/10/19 12:46:33 *************** *** 73,83 **** offset = count; if ( offset + size > count ) size = count - offset; ! ! /* if the base object is another buffer, then "deref" it */ ! if ( PyBuffer_Check(base) ) base = ((PyBufferObject *)base)->b_base; ! return _PyBuffer_FromMemory(base, (char *)p + offset, size, readonly); } --- 73,85 ---- offset = count; if ( offset + size > count ) size = count - offset; ! ! /* if the base object is another buffer, then "deref" it, ! * except if the base of the other buffer is NULL ! */ ! if ( PyBuffer_Check(base) && (((PyBufferObject *)base)->b_base) ) base = ((PyBufferObject *)base)->b_base; ! return _PyBuffer_FromMemory(base, (char *)p + offset, size, readonly); } From guido@python.org Fri Oct 19 14:31:28 2001 From: guido@python.org (Guido van Rossum) Date: Fri, 19 Oct 2001 09:31:28 -0400 Subject: [Python-Dev] Bug 216405 In-Reply-To: Your message of "Fri, 19 Oct 2001 14:49:40 +0200." <0ad801c1589c$8592fdf0$e000a8c0@thomasnotebook> References: <0ad801c1589c$8592fdf0$e000a8c0@thomasnotebook> Message-ID: <200110191331.JAA26393@cj20424-a.reston1.va.home.com> > I know most of you are busy with the release, but may I beg again > that SF bug #216405 be finally fixed in Python 2.2? > > I can apply the patch myself, if nobody has time, but > I need an OK from Guido ;-) OK. It's too late for the b1 release (Tim has already cooked the Windows installer) but it will go in the next releas. --Guido van Rossum (home page: http://www.python.org/~guido/) From thomas.heller@ion-tof.com Fri Oct 19 14:40:54 2001 From: thomas.heller@ion-tof.com (Thomas Heller) Date: Fri, 19 Oct 2001 15:40:54 +0200 Subject: [Python-Dev] Bug 216405 References: <0ad801c1589c$8592fdf0$e000a8c0@thomasnotebook> <200110191331.JAA26393@cj20424-a.reston1.va.home.com> Message-ID: <0c0a01c158a3$ad1a4ca0$e000a8c0@thomasnotebook> > > I know most of you are busy with the release, but may I beg again > > that SF bug #216405 be finally fixed in Python 2.2? > > > > I can apply the patch myself, if nobody has time, but > > I need an OK from Guido ;-) > > OK. It's too late for the b1 release (Tim has already cooked the > Windows installer) but it will go in the next releas. Great. I can live with the CVS version for now. Thanks, Thomas From fdrake@acm.org Fri Oct 19 14:52:58 2001 From: fdrake@acm.org (Fred L. Drake, Jr.) Date: Fri, 19 Oct 2001 09:52:58 -0400 Subject: [Python-Dev] Goals for patch selection for 2.1.2 In-Reply-To: <200110190617.IAA12488@pandora.informatik.hu-berlin.de> References: <200110190617.IAA12488@pandora.informatik.hu-berlin.de> Message-ID: <15312.12346.482959.64272@grendel.zope.com> Martin von Loewis writes: > One open issue would be to come up with a release schedule. Should > you restrict yourself to patches that also appear in 2.2 (perhaps in > a back-ported fashion), or will you give a deadline for new patches Since the point of a strictly bug-fix release is to fix bugs, I would not tie the 2.1.2 release to 2.2 in any way. The 2.2 developments remain a cheap source of bug fixes, though, so anything that was an actual bug fix should be considered for 2.1.2. If an alternate fix is needed because the codebase changed too much (which may be very possible this time), a completely new patch may be needed. -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> PythonLabs at Zope Corporation From guido@python.org Fri Oct 19 14:59:30 2001 From: guido@python.org (Guido van Rossum) Date: Fri, 19 Oct 2001 09:59:30 -0400 Subject: [Python-Dev] Goals for patch selection for 2.1.2 In-Reply-To: Your message of "Fri, 19 Oct 2001 14:31:37 +1000." <200110190431.f9J4VbL04554@mbuna.arbhome.com.au> References: <200110190431.f9J4VbL04554@mbuna.arbhome.com.au> Message-ID: <200110191359.JAA26790@cj20424-a.reston1.va.home.com> Thanks, Anthony! It's good to see someone volunteering without too much pressure. What's your SF id, so we can give you checkin permission? --Guido van Rossum (home page: http://www.python.org/~guido/) From barry@zope.com Fri Oct 19 15:13:59 2001 From: barry@zope.com (Barry A. Warsaw) Date: Fri, 19 Oct 2001 10:13:59 -0400 Subject: [Python-Dev] Has socketmodule compilation broken for anyone else lately? References: <116D27C8E12BD411B3AB00B0D022B0B87C46D2@yate.wa.CSIRO.AU> <200110191229.IAA26121@cj20424-a.reston1.va.home.com> Message-ID: <15312.13607.821357.720224@anthem.wooz.org> >>>>> "GvR" == Guido van Rossum <guido@python.org> writes: >> For the last couple of days, socketmodule hasn't compiled for >> me on Tru64 Unix. If I'm the only one seeing this, I'll file a >> bug report on sf... GvR> It's probably a missing \n\ in a string literal. I've fixed GvR> this on the release branch now. Thanks Guido. -Barry From fdrake@acm.org Fri Oct 19 15:14:59 2001 From: fdrake@acm.org (Fred L. Drake, Jr.) Date: Fri, 19 Oct 2001 10:14:59 -0400 Subject: [Python-Dev] Goals for patch selection for 2.1.2 In-Reply-To: <200110191359.JAA26790@cj20424-a.reston1.va.home.com> References: <200110190431.f9J4VbL04554@mbuna.arbhome.com.au> <200110191359.JAA26790@cj20424-a.reston1.va.home.com> Message-ID: <15312.13667.946275.569788@grendel.zope.com> Guido van Rossum writes: > Thanks, Anthony! It's good to see someone volunteering without too > much pressure. What's your SF id, so we can give you checkin > permission? "anthonybaxter". I've added him and set up permissions. Anthony, if I've left out any permissions, speak up and we'll get them fixed. Thanks for helping out! -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> PythonLabs at Zope Corporation From skip@pobox.com (Skip Montanaro) Fri Oct 19 16:47:48 2001 From: skip@pobox.com (Skip Montanaro) (Skip Montanaro) Date: Fri, 19 Oct 2001 10:47:48 -0500 Subject: [Python-Dev] should I check in this test_xmlrpc test case change? Message-ID: <15312.19236.314775.530443@beluga.mojam.com> When I made the latest change to xmlrpclib.py (to test for integer overflow), I also expanded the test_xmlrpc.py file. Unfortunately, I forgot to check it in. Should I do that now or file it as a separate patch? Skip From guido@python.org Fri Oct 19 16:50:13 2001 From: guido@python.org (Guido van Rossum) Date: Fri, 19 Oct 2001 11:50:13 -0400 Subject: [Python-Dev] should I check in this test_xmlrpc test case change? In-Reply-To: Your message of "Fri, 19 Oct 2001 10:47:48 CDT." <15312.19236.314775.530443@beluga.mojam.com> References: <15312.19236.314775.530443@beluga.mojam.com> Message-ID: <200110191550.LAA30349@cj20424-a.reston1.va.home.com> > When I made the latest change to xmlrpclib.py (to test for integer > overflow), I also expanded the test_xmlrpc.py file. Unfortunately, I forgot > to check it in. Should I do that now or file it as a separate patch? Assuming that it works for you :-), check it in to the trunk. It won't make it into 2.2b1, but it will be ready for 2.2b2. --Guido van Rossum (home page: http://www.python.org/~guido/) From barry@zope.com Fri Oct 19 16:54:29 2001 From: barry@zope.com (Barry A. Warsaw) Date: Fri, 19 Oct 2001 11:54:29 -0400 Subject: [Python-Dev] should I check in this test_xmlrpc test case change? References: <15312.19236.314775.530443@beluga.mojam.com> Message-ID: <15312.19637.671407.555401@anthem.wooz.org> >>>>> "SM" == Skip Montanaro <skip@pobox.com> writes: SM> When I made the latest change to xmlrpclib.py (to test for SM> integer overflow), I also expanded the test_xmlrpc.py file. SM> Unfortunately, I forgot to check it in. Should I do that now SM> or file it as a separate patch? You can check it into the trunk, but it's too late for anything else to make it in 2.2b1 now. -Barry From Martin.v.Loewis@t-online.de Fri Oct 19 16:56:00 2001 From: Martin.v.Loewis@t-online.de (Martin v. Loewis) Date: Fri, 19 Oct 2001 17:56:00 +0200 Subject: [Python-Dev] should I check in this test_xmlrpc test case change? Message-ID: <200110191556.f9JFu0J01595@mira.informatik.hu-berlin.de> > Should I do that now or file it as a separate patch? Please check it in now, so it appears in 2.2b2. Thanks, Martin From skip@pobox.com (Skip Montanaro) Fri Oct 19 17:02:27 2001 From: skip@pobox.com (Skip Montanaro) (Skip Montanaro) Date: Fri, 19 Oct 2001 11:02:27 -0500 Subject: [Python-Dev] should I check in this test_xmlrpc test case change? In-Reply-To: <15312.19637.671407.555401@anthem.wooz.org> References: <15312.19236.314775.530443@beluga.mojam.com> <15312.19637.671407.555401@anthem.wooz.org> Message-ID: <15312.20115.92879.794630@beluga.mojam.com> BAW> You can check it into the trunk, but it's too late for anything BAW> else to make it in 2.2b1 now. Thanks, will do. I didn't care about making 2.2b1 (it is just a test case, after all). I just didn't want to botch something up related to the mechanics of the release. Skip From barry@zope.com Fri Oct 19 17:07:25 2001 From: barry@zope.com (Barry A. Warsaw) Date: Fri, 19 Oct 2001 12:07:25 -0400 Subject: [Python-Dev] should I check in this test_xmlrpc test case change? References: <15312.19236.314775.530443@beluga.mojam.com> <15312.19637.671407.555401@anthem.wooz.org> <15312.20115.92879.794630@beluga.mojam.com> Message-ID: <15312.20413.437682.308810@anthem.wooz.org> >>>>> "SM" == Skip Montanaro <skip@pobox.com> writes: BAW> You can check it into the trunk, but it's too late for BAW> anything else to make it in 2.2b1 now. SM> Thanks, will do. I didn't care about making 2.2b1 (it is just SM> a test case, after all). I just didn't want to botch SM> something up related to the mechanics of the release. The trunk is always safe to make commits to. That's why we release from a branch. -Barry From barry@zope.com Fri Oct 19 18:00:26 2001 From: barry@zope.com (Barry A. Warsaw) Date: Fri, 19 Oct 2001 13:00:26 -0400 Subject: [Python-Dev] RELEASED: Python 2.2b1 is out! Message-ID: <15312.23594.31057.586571@anthem.wooz.org> We've released Python 2.2b1, the first beta release of Python 2.2, for your ebullience, ecdysis, and embarcation. As this is the first beta release, we are now in feature freeze for Python 2.2. http://www.python.org/2.2/ Give it a good try, and report what breaks to the bug tracker: http://sourceforge.net/bugs/?group_id=5470 New features in this release include: - New-style classes are now always dynamic, except for built-in and extension types - A very subtle syntactical pitfall in list comprehensions was fixed - getattr(obj, name, default) now only catches AttributeError - Weak reference objects are now part of the core and offers a C API. - unicode(obj) now behaves more like str(obj) - isinstance() now accepts more objects as its first and second arguments - os and posix supports chroot(), setgroups() and unsetenv() where available. The stat(), fstat(), statvfs() and fstatvfs() functions now return "pseudo-sequences" whose elements can be accessed by name or index. The time module's localtime(), gmtime() and strptime() are pseudo-sequences too. - Many improvements to the profiler As usual, Andrew Kuchling is writing a gentle introduction to the most important changes (currently excluding type/class unification), titled "What's New in Python 2.2": http://www.amk.ca/python/2.2/ There is an introduction to the type/class unification at: http://www.python.org/2.2/descrintro.html Thanks to everybody who contributed to this release, including all the alpha testers! brought-to-you-by-the-letter-"E"-ly y'rs, -Barry From fdrake@acm.org Fri Oct 19 21:46:13 2001 From: fdrake@acm.org (Fred L. Drake, Jr.) Date: Fri, 19 Oct 2001 16:46:13 -0400 Subject: [Python-Dev] Python documentation for iSilo Message-ID: <15312.37141.139330.26322@grendel.zope.com> I've re-worked the iSilo version of the documenation for the first beta of Python 2.2; each manual is now a separate document for iSilo, so you can pick & choose the documents you need. They are also smaller than the first test version (by about 1/3 of a megabyte), which is a nice savings. This version includes a few post-beta documentation fixes. To try it out, download the file isilo-2.2b1+.zip from: ftp://ftp.python.org/pub/python/doc/2.2b1/ Please send comments on this version of the docs to me at: python-docs@python.org Enjoy! -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> PythonLabs at Zope Corporation From fdrake@acm.org Fri Oct 19 21:59:04 2001 From: fdrake@acm.org (Fred L. Drake) Date: Fri, 19 Oct 2001 16:59:04 -0400 (EDT) Subject: [Python-Dev] [development doc updates] Message-ID: <20011019205904.1F95528697@beowolf.digicool.com> The development version of the documentation has been updated: http://python.sourceforge.net/devel-docs/ Documentation update beyond the Python 2.2 beta 1 docs. This matches the revised iSilo document conversion. From barry.alan.scott@ntlworld.com Fri Oct 19 23:54:28 2001 From: barry.alan.scott@ntlworld.com (Barry Scott) Date: Fri, 19 Oct 2001 23:54:28 +0100 Subject: [Python-Dev] Imports very slow on one NT4 system - any idea what to check for Message-ID: <000d01c158f1$020e9fc0$060210ac@private> I helping a friend get up to speed with Python. He has installed the ActiveState 2.1.1 on a number of machines. All run nice and fast except for one machine. It is one of group of identical desktop Dual 866MHz PIII systems with 1G RAM, SCSI hard drive running Windows NT4.0/SP6a. Importing getopt, re, sys, string and os takes 7 seconds. Using python -v just shows a uniform slowness, all files are slow. python -vv he tells me did not show a particular slow spot. His laptop Crusoe 700MHz machine, 192MB, IDE hard drive runs the same imports in 1.1 seconds. I've asked that he checks the python -v output to make sure that the .pyc files are being loaded and not the .py. Any idea what else to check for? Barry From davida@ActiveState.com Sat Oct 20 00:15:28 2001 From: davida@ActiveState.com (David Ascher) Date: Fri, 19 Oct 2001 16:15:28 -0700 (PDT) Subject: [Python-Dev] Imports very slow on one NT4 system - any idea what to check for In-Reply-To: <000d01c158f1$020e9fc0$060210ac@private> Message-ID: <Pine.LNX.4.30.0110191614320.19978-100000@latte.ActiveState.com> On Fri, 19 Oct 2001, Barry Scott wrote: > I helping a friend get up to speed with Python. He has installed the > ActiveState 2.1.1 on a number of machines. All run nice and fast except > for one machine. It is one of group of identical desktop Dual 866MHz PIII > systems with 1G RAM, SCSI hard drive running Windows NT4.0/SP6a. > > Importing getopt, re, sys, string and os takes 7 seconds. Using python -v > just shows a uniform slowness, all files are slow. python -vv he tells me > did not show a particular slow spot. > > His laptop Crusoe 700MHz machine, 192MB, IDE hard drive runs the same > imports in 1.1 seconds. > > I've asked that he checks the python -v output to make sure that the .pyc > files are being loaded and not the .py. > > Any idea what else to check for? This is probably not a good topic for python-dev. However, before I suggest that you take it up on python-list or activepython-feedback, let me suggest you check for virus-checking software. Some versions of Norton AntiVirus slow down file opens drastically. -- David Ascher From Anthony Baxter <anthony@interlink.com.au> Sun Oct 21 03:00:48 2001 From: Anthony Baxter <anthony@interlink.com.au> (Anthony Baxter) Date: Sun, 21 Oct 2001 12:00:48 +1000 Subject: [Python-Dev] Goals for patch selection for 2.1.2 In-Reply-To: Message from "Fred L. Drake, Jr." <fdrake@acm.org> of "Fri, 19 Oct 2001 09:52:58 -0400." <15312.12346.482959.64272@grendel.zope.com> Message-ID: <200110210201.f9L20mX29206@mbuna.arbhome.com.au> >>> "Fred L. Drake, Jr." wrote > Since the point of a strictly bug-fix release is to fix bugs, I > would not tie the 2.1.2 release to 2.2 in any way. The 2.2 > developments remain a cheap source of bug fixes, though, so anything > that was an actual bug fix should be considered for 2.1.2. If an > alternate fix is needed because the codebase changed too much (which > may be very possible this time), a completely new patch may be > needed. I agree - there's also the possibility that there will be additional patches needed to fix a bug in code that's just not relevant for 2.2 (this is less likely), or an alternate fix might be needed that's not as thorough as one in 2.2 (where there's a large fix, a smaller workaround might be more appropriate for 2.1.2) -- Anthony Baxter <anthony@interlink.com.au> It's never too late to have a happy childhood. From Anthony Baxter <anthony@interlink.com.au> Sun Oct 21 06:47:18 2001 From: Anthony Baxter <anthony@interlink.com.au> (Anthony Baxter) Date: Sun, 21 Oct 2001 15:47:18 +1000 Subject: [Python-Dev] Goals for patch selection for 2.1.2 In-Reply-To: Message from Guido van Rossum <guido@python.org> of "Fri, 19 Oct 2001 09:59:30 -0400." <200110191359.JAA26790@cj20424-a.reston1.va.home.com> Message-ID: <200110210547.f9L5lIP30307@mbuna.arbhome.com.au> Hm. What's the general consensus on portability fixes w.r.t. 2.1.2? I'm leaning toward "yes", but my concern is that they stand a chance of causing some other weird freakshow platform to fail to compile, where it previously worked... obviously I don't mean HP/UX threading here, as I think it's a lost cause :) But on the other hand, if it helps a platform build, where it previously did not... Hm hm hm. Anthony From tim.one@home.com Sun Oct 21 07:33:39 2001 From: tim.one@home.com (Tim Peters) Date: Sun, 21 Oct 2001 02:33:39 -0400 Subject: [Python-Dev] Goals for patch selection for 2.1.2 In-Reply-To: <200110210547.f9L5lIP30307@mbuna.arbhome.com.au> Message-ID: <LNBBLJKPBEHFEDALKOLCMEDOMCAA.tim.one@home.com> [Anthony Baxter] > Hm. What's the general consensus on portability fixes w.r.t. 2.1.2? If it's for a platform where Python used to run OK, it's probably "a bug" if 2.1.2 no longer runs on it (I say "probably" because you really don't want all the changes it would take to get Python working correctly under, e.g., WinCE again ...). If it's for a platform where Python has never run, then running on that platform is a new feature, and I wouldn't touch that in a bugfix release even if you stood between us and blocked most of the exploding guts <wink>. From aahz@rahul.net Sun Oct 21 15:35:48 2001 From: aahz@rahul.net (Aahz Maruch) Date: Sun, 21 Oct 2001 07:35:48 -0700 (PDT) Subject: [Python-Dev] Goals for patch selection for 2.1.2 In-Reply-To: <200110190431.f9J4VbL04554@mbuna.arbhome.com.au> from "Anthony Baxter" at Oct 19, 2001 02:31:37 PM Message-ID: <20011021143548.D66E7E8DF@waltz.rahul.net> Fred, could you update PEP 6 to make Anthony the 2.1.2 manager, or should I submit a patch to you? (Yeah, yeah, one of these days I'll actually do enough work to make it worth SF privileges.) Anthony Baxter wrote: > > Here's my current thoughts for selecting patches for a 2.1.2 release. > Feedback would be appreciated. I plan to post this to python-list and > ask people for suggestions for "must have" fixes. This could possibly > be added to PEP 006 once it's nailed down. I have zero objection to any single release using stricter guidelines than PEP 6 requires, but PEP 6 started life as a set of guidelines for both "bugfix" and "patch" releases. While I don't have a lot of objection to tightening up PEP 6 a bit more to reflect the fact that it focuses entirely on bugfix releases now, there are enough gray areas about what is a bug vs. feature that I don't want to make the PEP quite as strict as you plan to for 2.1.2. I don't want to get into a long discussion about this on python-dev; I'll respond to your post on c.l.py. I just wanted to make a short statement of principle in case anyone wants to get involved later. -- --- Aahz (@pobox.com) Hugs and backrubs -- I break Rule 6 <*> http://www.rahul.net/aahz/ Androgynous poly kinky vanilla queer het Pythonista We must not let the evil of a few trample the freedoms of the many. From BrainFletcher90@hotmail.com Sun Oct 21 17:27:57 2001 From: BrainFletcher90@hotmail.com (Brian Fletcher) Date: Sun, 21 Oct 2001 17:27:57 +0100 Subject: [Python-Dev] Fwd: Play a hilarious prank call Message-ID: <E15vLQQ-0004gI-00@mail.python.org> Computerized prank call service. Wind-up your mates! Click on this link. www.pranks.gq.nu (Works on any phone in the UK including mobiles) Or call 020 8387 8754 (24hrs) From tim.one@home.com Sun Oct 21 19:25:58 2001 From: tim.one@home.com (Tim Peters) Date: Sun, 21 Oct 2001 14:25:58 -0400 Subject: [Python-Dev] Goals for patch selection for 2.1.2 In-Reply-To: <20011021143548.D66E7E8DF@waltz.rahul.net> Message-ID: <LNBBLJKPBEHFEDALKOLCAEELMCAA.tim.one@home.com> [Aahz] > Fred, could you update PEP 6 to make Anthony the 2.1.2 manager, Done. i-am-my-fred's-keeper-ly y'rs - tim From guido@python.org Mon Oct 22 02:23:06 2001 From: guido@python.org (Guido van Rossum) Date: Sun, 21 Oct 2001 21:23:06 -0400 Subject: [Python-Dev] Goals for patch selection for 2.1.2 In-Reply-To: Your message of "Sun, 21 Oct 2001 15:47:18 +1000." <200110210547.f9L5lIP30307@mbuna.arbhome.com.au> References: <200110210547.f9L5lIP30307@mbuna.arbhome.com.au> Message-ID: <200110220123.VAA18017@cj20424-a.reston1.va.home.com> > obviously I don't mean HP/UX threading > here, as I think it's a lost cause :) Actually, reportedly 2.2b1 builds with threads and runs the thread tests correctly on HP-UX 11.00. See http://sf.net/tracker/?func=detail&atid=105470&aid=473150&group_id=5470 It's not clear what made this work though, so the backport may be muddy. You might try to enlist the help of the guy who reported that bug, he seems knowledgeable. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Mon Oct 22 02:33:38 2001 From: guido@python.org (Guido van Rossum) Date: Sun, 21 Oct 2001 21:33:38 -0400 Subject: [Python-Dev] Goals for patch selection for 2.1.2 In-Reply-To: Your message of "Sun, 21 Oct 2001 12:00:48 +1000." <200110210201.f9L20mX29206@mbuna.arbhome.com.au> References: <200110210201.f9L20mX29206@mbuna.arbhome.com.au> Message-ID: <200110220133.VAA19032@cj20424-a.reston1.va.home.com> > I agree - there's also the possibility that there will be additional > patches needed to fix a bug in code that's just not relevant for 2.2 > (this is less likely), or an alternate fix might be needed that's > not as thorough as one in 2.2 (where there's a large fix, a smaller > workaround might be more appropriate for 2.1.2) Yup -- this was the case for the profile.py patch (which took a life of its own in 2.2 :-). --Guido van Rossum (home page: http://www.python.org/~guido/) From barry@zope.com Mon Oct 22 06:36:38 2001 From: barry@zope.com (Barry A. Warsaw) Date: Mon, 22 Oct 2001 01:36:38 -0400 Subject: [Python-Dev] forwarded message from Guido van Rossum Message-ID: <15315.45158.289660.559852@anthem.wooz.org> With a cvs update, test_email.py fails because the behavior of re.split() has changed, presumably due to /F's latest sre checkins. Here's a boiled down example: -------------------- snip snip -------------------- import re ecre = re.compile(r''' =\? # literal =? (?P<charset>[^?]*?) # non-greedy up to the next ? is the charset \? # literal ? (?P<encoding>[qb]) # either a "q" or a "b", case insensitive \? # literal ? (?P<atom>.*?) # non-greedy up to the next ?= is the atom \?= # literal ?= ''', re.VERBOSE | re.IGNORECASE) print ecre.split('=?iso-8859-1?q?this=20is=20some=20text?=', 1) -------------------- snip snip -------------------- In Python 2.2b1 this prints ['', 'iso-8859-1', 'q', 'this=20is=20some=20text', ''] but in today's cvs this prints ['', 'iso-8859-1', 'q', 'this=20is=20some=20text'] Notice the missing trailing empty string. :( -Barry From tim.one@home.com Mon Oct 22 07:36:04 2001 From: tim.one@home.com (Tim Peters) Date: Mon, 22 Oct 2001 02:36:04 -0400 Subject: [Python-Dev] forwarded message from Guido van Rossum In-Reply-To: <15315.45158.289660.559852@anthem.wooz.org> Message-ID: <LNBBLJKPBEHFEDALKOLCOEGCMCAA.tim.one@home.com> [Barry] > With a cvs update, test_email.py fails because the behavior of > re.split() has changed, presumably due to /F's latest sre checkins. I believe /F has since fixed this (and test_email.py works for me again). From barry@zope.com Mon Oct 22 07:35:46 2001 From: barry@zope.com (Barry A. Warsaw) Date: Mon, 22 Oct 2001 02:35:46 -0400 Subject: [Python-Dev] Re: [Python-checkins] CVS: python/dist/src/Modules _sre.c,2.71,2.72 References: <E15vY9u-0003L6-00@usw-pr-cvs1.sourceforge.net> Message-ID: <15315.48706.655213.385913@anthem.wooz.org> >>>>> "FL" == Fredrik Lundh <effbot@users.sourceforge.net> writes: FL> sre.split should return the last segment, even if empty FL> (sorry, barry) Thank you effbot! From loewis@informatik.hu-berlin.de Mon Oct 22 09:02:35 2001 From: loewis@informatik.hu-berlin.de (Martin von Loewis) Date: Mon, 22 Oct 2001 10:02:35 +0200 (MEST) Subject: [Python-Dev] Goals for patch selection for 2.1.2 In-Reply-To: <200110210201.f9L20mX29206@mbuna.arbhome.com.au> (message from Anthony Baxter on Sun, 21 Oct 2001 12:00:48 +1000) References: <200110210201.f9L20mX29206@mbuna.arbhome.com.au> Message-ID: <200110220802.KAA27251@paros.informatik.hu-berlin.de> > I agree - there's also the possibility that there will be additional > patches needed to fix a bug in code that's just not relevant for 2.2 > (this is less likely), or an alternate fix might be needed that's > not as thorough as one in 2.2 (where there's a large fix, a smaller > workaround might be more appropriate for 2.1.2) Then I come back to my original question: What is your schedule for 2.1.2 then? I don't mind if it is either very tight or very lose, but I think there should be a schedule (which can be adjusted as needed). I think the following milestones should appear: - deadline for contributing new patches - deadline for selecting and integrating CVS mainline patches - release of a release candidate - release IMO, it isn't important that *all* patches that are desirable for 2.1.2 really get integrated by the second deadline. Instead, what gets done is done, everything else is for 2.1.3 (if there is interest in such a release). E.g. I wouldn't delay the release indefinitely just because somebody promised to indicate all, say, pyexpat patches, and never did. Regards, Martin P.S. Although I certainly do promise to indicate all pyexpat patches that should go into 2.1.2 :-) From Anthony Baxter <anthony@interlink.com.au> Mon Oct 22 15:20:34 2001 From: Anthony Baxter <anthony@interlink.com.au> (Anthony Baxter) Date: Tue, 23 Oct 2001 00:20:34 +1000 Subject: [Python-Dev] Goals for patch selection for 2.1.2 In-Reply-To: Message from Martin von Loewis <loewis@informatik.hu-berlin.de> of "Mon, 22 Oct 2001 10:02:35 +0200." <200110220802.KAA27251@paros.informatik.hu-berlin.de> Message-ID: <200110221420.f9MEKYc04949@mbuna.arbhome.com.au> >>> Martin von Loewis wrote > Then I come back to my original question: What is your schedule for > 2.1.2 then? Darn. Was hoping you'd missed that I'd put that question off :) At this point, I'm still trawling CVS - I hope to have an estimate in a day or so (work stuff permitting), or the end of the week at the latest. My gut feel right now is that it's a couple of weeks to finish trawling the CVS logs (geez you guys commit a lot of changes) and checking them in carefully). > I don't mind if it is either very tight or very lose, but > I think there should be a schedule (which can be adjusted as > needed). I think the following milestones should appear: > - deadline for contributing new patches > - deadline for selecting and integrating CVS mainline patches > - release of a release candidate > - release Sounds like a good set of milestones. As I said, though, right now I'm just not sure. > IMO, it isn't important that *all* patches that are desirable for > 2.1.2 really get integrated by the second deadline. Instead, what gets Nope. I'd like to be able to say, though, that "for this release, there are some known bugs in pyexpat/the floating point overflow/ whatever" - at least let people know. > P.S. Although I certainly do promise to indicate all pyexpat patches > that should go into 2.1.2 :-) I'm glad to hear it :) Anthony -- Anthony Baxter <anthony@interlink.com.au> It's never too late to have a happy childhood. From loewis@informatik.hu-berlin.de Mon Oct 22 15:36:04 2001 From: loewis@informatik.hu-berlin.de (Martin von Loewis) Date: Mon, 22 Oct 2001 16:36:04 +0200 (MEST) Subject: [Python-Dev] Goals for patch selection for 2.1.2 In-Reply-To: <200110221420.f9MEKYc04949@mbuna.arbhome.com.au> (message from Anthony Baxter on Tue, 23 Oct 2001 00:20:34 +1000) References: <200110221420.f9MEKYc04949@mbuna.arbhome.com.au> Message-ID: <200110221436.QAA27751@paros.informatik.hu-berlin.de> > Darn. Was hoping you'd missed that I'd put that question off :) > At this point, I'm still trawling CVS - I hope to have an estimate > in a day or so (work stuff permitting), or the end of the week at > the latest. My gut feel right now is that it's a couple of weeks > to finish trawling the CVS logs (geez you guys commit a lot of > changes) and checking them in carefully). Remember you are Patch Czar, not the slave of patches. You could restrict yourself to incorporate only patches that people propose. If you want systematic scanning, we could try to come up with a script that automatically reviews all commit messages of patches since 2.1.1 whether these mention an SF bug. We could then further automate scanning to produce the number of changed lines, to exclude 'large' changes. Perhaps anybody in the readership of this software already has code that does such things? Regards, Martin From Anthony Baxter <anthony@interlink.com.au> Mon Oct 22 15:39:25 2001 From: Anthony Baxter <anthony@interlink.com.au> (Anthony Baxter) Date: Tue, 23 Oct 2001 00:39:25 +1000 Subject: [Python-Dev] Goals for patch selection for 2.1.2 In-Reply-To: Message from Martin von Loewis <loewis@informatik.hu-berlin.de> of "Mon, 22 Oct 2001 16:36:04 +0200." <200110221436.QAA27751@paros.informatik.hu-berlin.de> Message-ID: <200110221439.f9MEdP105195@mbuna.arbhome.com.au> > Remember you are Patch Czar, not the slave of patches. You could > restrict yourself to incorporate only patches that people propose. Heh. No, the problem at this point as far as making estimates go is even knowing a vague idea of how much there is to do. > If you want systematic scanning, we could try to come up with a script > that automatically reviews all commit messages of patches since 2.1.1 > whether these mention an SF bug. We could then further automate > scanning to produce the number of changed lines, to exclude 'large' > changes. Hrm. That might be an idea - at the moment I've got the trunk and the 2.1 branch checked out in parallel directories, and have a teeny zsh alias to do diffs and look at the tree. I'm then checking the commit logs since the 2.1 branch to see the diffs. > Perhaps anybody in the readership of this software already has code > that does such things? We can hope... -- Anthony Baxter <anthony@interlink.com.au> It's never too late to have a happy childhood. From guido@python.org Mon Oct 22 15:41:16 2001 From: guido@python.org (Guido van Rossum) Date: Mon, 22 Oct 2001 10:41:16 -0400 Subject: [Python-Dev] Goals for patch selection for 2.1.2 In-Reply-To: Your message of "Mon, 22 Oct 2001 16:36:04 +0200." <200110221436.QAA27751@paros.informatik.hu-berlin.de> References: <200110221420.f9MEKYc04949@mbuna.arbhome.com.au> <200110221436.QAA27751@paros.informatik.hu-berlin.de> Message-ID: <200110221441.KAA30687@cj20424-a.reston1.va.home.com> > If you want systematic scanning, we could try to come up with a script > that automatically reviews all commit messages of patches since 2.1.1 > whether these mention an SF bug. We could then further automate > scanning to produce the number of changed lines, to exclude 'large' > changes. Slightly dangerous though -- some bugfixes were never reported to SF. > Perhaps anybody in the readership of this software already has code > that does such things? No, but you can start with Tools/scripts/logmerge.py which parses CVS log output and sorts it by date. The best way to scan for SF patch refs is probably to search for 6-digit numbers... There's too much diversity otherwise in the SF references :-( --Guido van Rossum (home page: http://www.python.org/~guido/) From aahz@rahul.net Mon Oct 22 16:41:10 2001 From: aahz@rahul.net (Aahz Maruch) Date: Mon, 22 Oct 2001 08:41:10 -0700 (PDT) Subject: [Python-Dev] Goals for patch selection for 2.1.2 In-Reply-To: <200110221436.QAA27751@paros.informatik.hu-berlin.de> from "Martin von Loewis" at Oct 22, 2001 04:36:04 PM Message-ID: <20011022154110.6E372E8C1@waltz.rahul.net> Martin von Loewis wrote: > > Remember you are Patch Czar, not the slave of patches. You could > restrict yourself to incorporate only patches that people propose. > > If you want systematic scanning, we could try to come up with a script > that automatically reviews all commit messages of patches since 2.1.1 > whether these mention an SF bug. We could then further automate > scanning to produce the number of changed lines, to exclude 'large' > changes. As it says in PEP 6, one thing I was hoping for was that people submitting patches would indicate whether a patch ought to be considered for inclusion in patch releases for prior formal releases. One way to do that would be for people to check in patches to the prior version's trunk, and then the Patch Czar could decide which patches belong in a release branch. Of course, that's assuming I understand how CVS works, which is probably false. ;-) Alternatively, we could agree on some text tag format for use in SF comments. -- --- Aahz (@pobox.com) Hugs and backrubs -- I break Rule 6 <*> http://www.rahul.net/aahz/ Androgynous poly kinky vanilla queer het Pythonista We must not let the evil of a few trample the freedoms of the many. From aahz@rahul.net Mon Oct 22 16:47:00 2001 From: aahz@rahul.net (Aahz Maruch) Date: Mon, 22 Oct 2001 08:47:00 -0700 (PDT) Subject: [Python-Dev] Goals for patch selection for 2.1.2 In-Reply-To: <200110220802.KAA27251@paros.informatik.hu-berlin.de> from "Martin von Loewis" at Oct 22, 2001 10:02:35 AM Message-ID: <20011022154700.A0C65E8C1@waltz.rahul.net> Martin von Loewis wrote: > > I think the following milestones should appear: > - deadline for contributing new patches > - deadline for selecting and integrating CVS mainline patches > - release of a release candidate > - release I can't remember whether it was Tim or Guido, but it has been suggested that a true bugfix release shouldn't need a release candidate. Me, I'm always in favor of release candidates. [skip this next bit if you don't want an anecdote] (Back when I knew less about Unix than I do now, I was two weeks into a new job, dealing with a customer who was complaining about being unable to install the latest release of our software. I tracked it down to a syntax error in the install shell script -- proving that *NOBODY* from QA had bothered to do an actual test install with the released CD.) -- --- Aahz (@pobox.com) Hugs and backrubs -- I break Rule 6 <*> http://www.rahul.net/aahz/ Androgynous poly kinky vanilla queer het Pythonista We must not let the evil of a few trample the freedoms of the many. From barry@zope.com Mon Oct 22 16:58:14 2001 From: barry@zope.com (Barry A. Warsaw) Date: Mon, 22 Oct 2001 11:58:14 -0400 Subject: [Python-Dev] Goals for patch selection for 2.1.2 References: <200110220802.KAA27251@paros.informatik.hu-berlin.de> <20011022154700.A0C65E8C1@waltz.rahul.net> Message-ID: <15316.16918.889688.286867@anthem.wooz.org> >>>>> "AM" == Aahz Maruch <aahz@rahul.net> writes: AM> I can't remember whether it was Tim or Guido, but it has been AM> suggested that a true bugfix release shouldn't need a release AM> candidate. Me, I'm always in favor of release candidates. Possibly me, because I don't think release candidates on micro-releases are that important. If you royally screwed up 2.1.2 it should be low-cost to turnaround and do another 2.1.3 release very quickly. I'd rather opt for quick, low-overhead micro/patch releases. -Barry From fdrake@acm.org Mon Oct 22 18:24:35 2001 From: fdrake@acm.org (Fred L. Drake) Date: Mon, 22 Oct 2001 13:24:35 -0400 (EDT) Subject: [Python-Dev] [development doc updates] Message-ID: <20011022172435.28AF528697@cj42289-a.reston1.va.home.com> The development version of the documentation has been updated: http://python.sourceforge.net/devel-docs/ Various updates, including support for the "Site Navigation Bar" in Mozilla 0.9.5. From tim.one@home.com Mon Oct 22 19:08:32 2001 From: tim.one@home.com (Tim Peters) Date: Mon, 22 Oct 2001 14:08:32 -0400 Subject: [Python-Dev] Goals for patch selection for 2.1.2 In-Reply-To: <200110221420.f9MEKYc04949@mbuna.arbhome.com.au> Message-ID: <LNBBLJKPBEHFEDALKOLCCEHLMCAA.tim.one@home.com> [Anthony Baxter] > Darn. Was hoping you'd missed that I'd put that question off :) > ... Aren't you glad you asked an innocent question about bugfix releases on c.l.py? Aren't you glad I replied? Heh heh -- that's how all Python work gets done in the end <wink>. tag-you're-it-ly y'rs - tim From tim.one@home.com Mon Oct 22 21:10:36 2001 From: tim.one@home.com (Tim Peters) Date: Mon, 22 Oct 2001 16:10:36 -0400 Subject: [Python-Dev] Goals for patch selection for 2.1.2 In-Reply-To: <20011022154700.A0C65E8C1@waltz.rahul.net> Message-ID: <LNBBLJKPBEHFEDALKOLCOEIHMCAA.tim.one@home.com> [Aahz] > ... > (Back when I knew less about Unix than I do now, I was two weeks into a > new job, dealing with a customer who was complaining about being unable > to install the latest release of our software. I tracked it down to a > syntax error in the install shell script -- proving that *NOBODY* from > QA had bothered to do an actual test install with the released CD.) In my Cray Research days, releases were shipped on magnetic tape. A day after one major release, customers started to report that the tapes they got were blank -- the tape unit write head was in fact utterly dead, but nobody had bothered to even trying reading the release media, let alone install from it. PythonLabs has also shipped damaged tarballs. The only thing we haven't done is ship an empty Windows installer <wink>. From DavidA@ActiveState.com Mon Oct 22 21:46:50 2001 From: DavidA@ActiveState.com (David Ascher) Date: 22 Oct 2001 13:46:50 -0700 Subject: [Python-Dev] Goals for patch selection for 2.1.2 In-Reply-To: <LNBBLJKPBEHFEDALKOLCOEIHMCAA.tim.one@home.com> References: <LNBBLJKPBEHFEDALKOLCOEIHMCAA.tim.one@home.com> Message-ID: <1003783610.23693.44.camel@loom.activestate.com> On Mon, 2001-10-22 at 13:10, Tim Peters wrote: > [Aahz] > > ... > > (Back when I knew less about Unix than I do now, I was two weeks into a > > new job, dealing with a customer who was complaining about being unable > > to install the latest release of our software. I tracked it down to a > > syntax error in the install shell script -- proving that *NOBODY* from > > QA had bothered to do an actual test install with the released CD.) > > In my Cray Research days, releases were shipped on magnetic tape. A day > after one major release, customers started to report that the tapes they got > were blank -- the tape unit write head was in fact utterly dead, but nobody > had bothered to even trying reading the release media, let alone install > from it. > > PythonLabs has also shipped damaged tarballs. The only thing we haven't > done is ship an empty Windows installer <wink>. Myself, I'll never admit anything in public. --david From tim.one@home.com Mon Oct 22 21:56:42 2001 From: tim.one@home.com (Tim Peters) Date: Mon, 22 Oct 2001 16:56:42 -0400 Subject: [Python-Dev] Goals for patch selection for 2.1.2 In-Reply-To: <1003783610.23693.44.camel@loom.activestate.com> Message-ID: <LNBBLJKPBEHFEDALKOLCAEILMCAA.tim.one@home.com> [David Ascher] > Myself, I'll never admit anything in public. Of course, but we have to: it's the difference between working for an Open Source company and working for Microsoft <wink>. From DavidA@ActiveState.com Mon Oct 22 22:02:35 2001 From: DavidA@ActiveState.com (David Ascher) Date: 22 Oct 2001 14:02:35 -0700 Subject: [Python-Dev] Goals for patch selection for 2.1.2 In-Reply-To: <LNBBLJKPBEHFEDALKOLCAEILMCAA.tim.one@home.com> References: <LNBBLJKPBEHFEDALKOLCAEILMCAA.tim.one@home.com> Message-ID: <1003784555.23700.52.camel@loom.activestate.com> On Mon, 2001-10-22 at 13:56, Tim Peters wrote: > [David Ascher] > > Myself, I'll never admit anything in public. > > Of course, but we have to: it's the difference between working for an Open > Source company and working for Microsoft <wink>. Yowza! If I were working for Microsoft, I wouldn't be dealing with GTK+ issues these days =). --david From jeremy@zope.com Tue Oct 23 01:52:09 2001 From: jeremy@zope.com (Jeremy Hylton) Date: Mon, 22 Oct 2001 20:52:09 -0400 (EDT) Subject: [Python-Dev] Goals for patch selection for 2.1.2 In-Reply-To: <200110190431.f9J4VbL04554@mbuna.arbhome.com.au> References: <200110190431.f9J4VbL04554@mbuna.arbhome.com.au> Message-ID: <15316.48953.976319.490902@slothrop.digicool.com> >>>>> "AB" == Anthony Baxter <anthony@interlink.com.au> writes: AB> Things that shouldn't be considered: [...] AB> Fixes that "remove a nasty workaround" AB> In general, I'd like the starting point for any patch to be AB> considered as "guilty until proven necessary". Do you have an opinion on the recent change to traceback.c (make it collectable)? I think it's a good candidate for 2.1.2 even though it doesn't meet the criteria above :-). It removes the nasty workaround that you *never* exit a function with a local variable that points to a traceback that references the function. There's a bit of C code (most of it probably extending the type structure for tracebacks), but it's simple stuff, thoroughly tested by the current test suite, and it's a *nasty* problem in practice. Two other things I thought of on this thread: We got out of the habit of marking changes in 2.2 as potential bug fixes, because we didn't expect to have a 2.1.2 release. I think we should continue marking checkins this way indefinitely. We may not have a 2.1.3 and a 2.2.2, but it's pretty easy to mark the checkin just in case we do make such a release. I'd like to update the compiler package in Tools/compiler, because I've fixed a number of nasty bugs, e.g. computing a stack size that is too small for a function. But I don't know if I'll have time. Once you've got a schedule, I'll let you know if I can manage it. Jeremy From guido@python.org Tue Oct 23 02:14:46 2001 From: guido@python.org (Guido van Rossum) Date: Mon, 22 Oct 2001 21:14:46 -0400 Subject: [Python-Dev] Goals for patch selection for 2.1.2 In-Reply-To: Your message of "Mon, 22 Oct 2001 20:52:09 EDT." <15316.48953.976319.490902@slothrop.digicool.com> References: <200110190431.f9J4VbL04554@mbuna.arbhome.com.au> <15316.48953.976319.490902@slothrop.digicool.com> Message-ID: <200110230114.VAA03866@cj20424-a.reston1.va.home.com> > AB> In general, I'd like the starting point for any patch to be > AB> considered as "guilty until proven necessary". > > Do you have an opinion on the recent change to traceback.c (make it > collectable)? I think it's a good candidate for 2.1.2 even though it > doesn't meet the criteria above :-). Hm. In 2.2, the C API for the garbage collector is different from 2.1, so backporting this would require extra work. Also, you'd probably have to add GC to frames as well to reap the benefits. To me this particular problem feels like a bug that's almost a feature (everybody knows about it, it's documented, etc.), and I think we'd be better off not fixing this in 2.1.2. --Guido van Rossum (home page: http://www.python.org/~guido/) From tim.one@home.com Tue Oct 23 06:05:59 2001 From: tim.one@home.com (Tim Peters) Date: Tue, 23 Oct 2001 01:05:59 -0400 Subject: [Python-Dev] Goals for patch selection for 2.1.2 In-Reply-To: <15316.48953.976319.490902@slothrop.digicool.com> Message-ID: <LNBBLJKPBEHFEDALKOLCKEKDMCAA.tim.one@home.com> [Jeremy] > Do you have an opinion on the recent change to traceback.c (make it > collectable)? I think it's a good candidate for 2.1.2 even though it > doesn't meet the criteria above :-). Guido's right: that change alone won't do any good. Frame objects need to be chased by gc too, else the cycle would survive. However, adding frame objects to gc caused a measurable slowdown, which I judged tolerable in 2.2 because 2.2 got enough non-bugfix speedup work to cancel it out (at the time -- I expect we're losing again by now). if-it-ain't-a-pure-self-contained-bugfix-it's-dubious-at-best-ly y'rs - tim From akuchlin@mems-exchange.org Tue Oct 23 14:11:26 2001 From: akuchlin@mems-exchange.org (akuchlin@mems-exchange.org) Date: Tue, 23 Oct 2001 09:11:26 -0400 Subject: [Python-Dev] "Leaving Sourceforge" article on Advogato Message-ID: <20011023091126.A24293@mems-exchange.org> There's a discussion of SourceForge alternatives at http://www.advogato.org/article/357.html, and in a comment someone mentions that there's an XML export of the data in the tracker. Unfortunately I can't get it to work following his instructions, but perhaps someone else can experiment with it more than I can. --amk From barry@zope.com Tue Oct 23 15:02:34 2001 From: barry@zope.com (Barry A. Warsaw) Date: Tue, 23 Oct 2001 10:02:34 -0400 Subject: [Python-Dev] "Leaving Sourceforge" article on Advogato References: <20011023091126.A24293@mems-exchange.org> Message-ID: <15317.30842.636138.922307@anthem.wooz.org> >>>>> "akuchlin" == <akuchlin@mems-exchange.org> writes: akuchlin> There's a discussion of SourceForge alternatives at akuchlin> http://www.advogato.org/article/357.html, and in a akuchlin> comment someone mentions that there's an XML export of akuchlin> the data in the tracker. Unfortunately I can't get it akuchlin> to work following his instructions, but perhaps someone akuchlin> else can experiment with it more than I can. I can't get it to work either, but if someone comes up with the right recipe, I can add it to our backup script. Every night I pull down the CVS repositories for Python, Mailman, Jython, and mimelib. I also grab the Jython FAQ and moinmoin. These I backup nightly on a Pythonlabs/ZC server. So I feel fairly confident about this data. But I'd love to find a solution for the tracker data too. -Barry From fredrik@pythonware.com Tue Oct 23 16:35:44 2001 From: fredrik@pythonware.com (Fredrik Lundh) Date: Tue, 23 Oct 2001 17:35:44 +0200 Subject: [Python-Dev] "Leaving Sourceforge" article on Advogato References: <20011023091126.A24293@mems-exchange.org> <15317.30842.636138.922307@anthem.wooz.org> Message-ID: <003b01c15bd8$6cffede0$ced241d5@hagrid> barry wrote: > akuchlin> There's a discussion of SourceForge alternatives at > akuchlin> http://www.advogato.org/article/357.html, and in a > akuchlin> comment someone mentions that there's an XML export of > akuchlin> the data in the tracker. Unfortunately I can't get it > akuchlin> to work following his instructions, but perhaps someone > akuchlin> else can experiment with it more than I can. > > I can't get it to work either, but if someone comes up with the right > recipe, I can add it to our backup script. works fine from MSIE, using this URL: url = ( "http://sourceforge.net/export/sf_tracker_export.php" + "?atid=105470&group_id=5470" ) returns a 1666445 byte large document (with no content type), starting with: <tracker version="1.0" xmlns:xsi="http://www.w3.org/2000/10/XMLSchema-instance" xsi:noNamespaceSchemaLocation="http://sourceforge.net/export/sf_tracker_export.xsd"> <artifact id="473985"> <submitted_by>doerwalter</submitted_by> <submitted_date>2001-10-23 03:39</submitted_date> <artifact_type id="105470">105470</artifact_type> <category id="347131">Type/class unification</category> <artifact_group id="136128"></artifact_group> <assigned_to>Guido van Rossum</assigned_to> <priority id="5">5</priority> <status>Open</status> <resolution>Remind</resolution> <summary>str, __getitem__ and slices</summary> <detail>Using slices with __getitem__ doesn't work with (this returns the bugs database. see http://sourceforge.net/tracker/?group_id=5470 for atid's for other parts of the tracker) doing the same through urllib doesn't appear work, though. (no time to dig further into this right now) </F> From aahz@rahul.net Tue Oct 23 16:35:04 2001 From: aahz@rahul.net (Aahz Maruch) Date: Tue, 23 Oct 2001 08:35:04 -0700 (PDT) Subject: [Python-Dev] "Leaving Sourceforge" article on Advogato In-Reply-To: <003b01c15bd8$6cffede0$ced241d5@hagrid> from "Fredrik Lundh" at Oct 23, 2001 05:35:44 PM Message-ID: <20011023153504.D66C6E8C5@waltz.rahul.net> Fredrik Lundh wrote: > > works fine from MSIE, using this URL: > [....] > doing the same through urllib doesn't appear work, though. Well, that's a good reason for leaving SF. :-/ -- --- Aahz (@pobox.com) Hugs and backrubs -- I break Rule 6 <*> http://www.rahul.net/aahz/ Androgynous poly kinky vanilla queer het Pythonista We must not let the evil of a few trample the freedoms of the many. From loewis@informatik.hu-berlin.de Tue Oct 23 18:25:30 2001 From: loewis@informatik.hu-berlin.de (Martin von Loewis) Date: Tue, 23 Oct 2001 19:25:30 +0200 (MEST) Subject: [Python-Dev] "Leaving Sourceforge" article on Advogato Message-ID: <200110231725.TAA11514@paros.informatik.hu-berlin.de> > doing the same through urllib doesn't appear work, though. Nor does it work with Konqueror or Netscape, see http://sourceforge.net/tracker/?func=detail&aid=469987&group_id=1&atid=100001 It's good to know it works with IE, though. Regards, Martin From mal@lemburg.com Tue Oct 23 19:28:55 2001 From: mal@lemburg.com (M.-A. Lemburg) Date: Tue, 23 Oct 2001 20:28:55 +0200 Subject: [Python-Dev] "Leaving Sourceforge" article on Advogato References: <20011023091126.A24293@mems-exchange.org> <15317.30842.636138.922307@anthem.wooz.org> <003b01c15bd8$6cffede0$ced241d5@hagrid> Message-ID: <3BD5B6E7.CF511C1@lemburg.com> Fredrik Lundh wrote: > > barry wrote: > > > akuchlin> There's a discussion of SourceForge alternatives at > > akuchlin> http://www.advogato.org/article/357.html, and in a > > akuchlin> comment someone mentions that there's an XML export of > > akuchlin> the data in the tracker. Unfortunately I can't get it > > akuchlin> to work following his instructions, but perhaps someone > > akuchlin> else can experiment with it more than I can. > > > > I can't get it to work either, but if someone comes up with the right > > recipe, I can add it to our backup script. > > works fine from MSIE, using this URL: > > url = ( > "http://sourceforge.net/export/sf_tracker_export.php" + > "?atid=105470&group_id=5470" > ) > > returns a 1666445 byte large document (with no content > type), starting with: > ... > (this returns the bugs database. see > http://sourceforge.net/tracker/?group_id=5470 > for atid's for other parts of the tracker) > > doing the same through urllib doesn't appear work, though. > > (no time to dig further into this right now) FYI, It seems that you have to be logged into SF in order for the download to work. Netscape does the download too if you create a web page and then use the right-click-download feature. -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Consulting & Company: http://www.egenix.com/ Python Software: http://www.lemburg.com/python/ From akuchlin@mems-exchange.org Tue Oct 23 21:13:29 2001 From: akuchlin@mems-exchange.org (Andrew Kuchling) Date: 23 Oct 2001 20:13:29 -0000 Subject: [Python-Dev] Dictionary subclasses and exec Message-ID: <20011023201329.27143.qmail@mozart.cnri.reston.va.us> The following bit of code defines a dictionary that folds all its keys to lowercase before inserting them, but 'exec "Name = 1" in LowerCaseDict()' doesn't seem to call my __getitem__(), as the listed keys include 'Name'. Should this be expected to work? --amk class LowerCaseDict(dictionary): def _fold_key (self, key): if not isinstance(key, str): raise TypeError, "All keys must be strings" return key.lower() def __getitem__ (self, key): key = self._fold_key(key) return dictionary.__getitem__(self, key) def __setitem__ (self, key, value): key = self._fold_key(key) dictionary.__setitem__(self, key, value) def __delitem__ (self, key): key = self._fold_key(key) dictionary.__delitem__(self, key, value) d = LowerCaseDict() exec 'Name = 1' in d print d.keys() From guido@python.org Tue Oct 23 22:08:15 2001 From: guido@python.org (Guido van Rossum) Date: Tue, 23 Oct 2001 17:08:15 -0400 Subject: [Python-Dev] Dictionary subclasses and exec In-Reply-To: Your message of "23 Oct 2001 20:13:29 -0000." <20011023201329.27143.qmail@mozart.cnri.reston.va.us> References: <20011023201329.27143.qmail@mozart.cnri.reston.va.us> Message-ID: <200110232108.f9NL8Fi25565@odiug.zope.com> > The following bit of code defines a dictionary that folds all its keys > to lowercase before inserting them, but 'exec "Name = 1" in > LowerCaseDict()' doesn't seem to call my __getitem__(), as the listed > keys include 'Name'. Should this be expected to work? > > --amk > > class LowerCaseDict(dictionary): > def _fold_key (self, key): > if not isinstance(key, str): > raise TypeError, "All keys must be strings" > return key.lower() > > def __getitem__ (self, key): > key = self._fold_key(key) > return dictionary.__getitem__(self, key) > > def __setitem__ (self, key, value): > key = self._fold_key(key) > dictionary.__setitem__(self, key, value) > > def __delitem__ (self, key): > key = self._fold_key(key) > dictionary.__delitem__(self, key, value) > > d = LowerCaseDict() > exec 'Name = 1' in d > print d.keys() Alas, this is one of the things that don't work yet. To set and get local variables, exec uses lower-level APIs (PyDict_SetItem and PyDict_GetItem) that you can't override. I've thought about what it would take to make this work as desired, but I haven't found a way yet that wouldn't (a) slow down the normal case, or (b) create subtle reference count bugs. --Guido van Rossum (home page: http://www.python.org/~guido/) From akuchlin@mems-exchange.org Tue Oct 23 23:17:37 2001 From: akuchlin@mems-exchange.org (akuchlin@mems-exchange.org) Date: Tue, 23 Oct 2001 18:17:37 -0400 Subject: [Python-Dev] Dictionary subclasses and exec In-Reply-To: <200110232108.f9NL8Fi25565@odiug.zope.com>; from guido@python.org on Tue, Oct 23, 2001 at 05:08:15PM -0400 References: <20011023201329.27143.qmail@mozart.cnri.reston.va.us> <200110232108.f9NL8Fi25565@odiug.zope.com> Message-ID: <20011023181737.B27568@mems-exchange.org> On Tue, Oct 23, 2001 at 05:08:15PM -0400, Guido van Rossum wrote: >Alas, this is one of the things that don't work yet. To set and get Oh, well; I need to come up with a different example for subclassing a built-in type, then. Thanks! <puts on thinking cap> --amk From guido@python.org Wed Oct 24 00:58:27 2001 From: guido@python.org (Guido van Rossum) Date: Tue, 23 Oct 2001 19:58:27 -0400 Subject: [Python-Dev] Dictionary subclasses and exec In-Reply-To: Your message of "Tue, 23 Oct 2001 18:17:37 EDT." <20011023181737.B27568@mems-exchange.org> References: <20011023201329.27143.qmail@mozart.cnri.reston.va.us> <200110232108.f9NL8Fi25565@odiug.zope.com> <20011023181737.B27568@mems-exchange.org> Message-ID: <200110232358.TAA06401@cj20424-a.reston1.va.home.com> > >Alas, this is one of the things that don't work yet. To set and get > > Oh, well; I need to come up with a different example for subclassing a > built-in type, then. Thanks! <puts on thinking cap> You can use a similar example but not override __getitem__ / __setitem__; instead, add some new methods (e.g. merge() -- like update() but the existing key wins). --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Wed Oct 24 01:00:44 2001 From: guido@python.org (Guido van Rossum) Date: Tue, 23 Oct 2001 20:00:44 -0400 Subject: [Python-Dev] patchlevel.h In-Reply-To: Your message of "Tue, 23 Oct 2001 15:21:00 PDT." <E15w9uu-0004HJ-00@usw-pr-cvs1.sourceforge.net> References: <E15w9uu-0004HJ-00@usw-pr-cvs1.sourceforge.net> Message-ID: <200110240000.UAA06439@cj20424-a.reston1.va.home.com> Jack, you can't do this on the trunk. If you *have* to have a different patch level, please use a branch! --Guido van Rossum (home page: http://www.python.org/~guido/) > Update of /cvsroot/python/python/dist/src/Include > In directory usw-pr-cvs1:/tmp/cvs-serv16428/Python/Include > > Modified Files: > patchlevel.h > Log Message: > Tweaks for MacPython 2.2b1 > > Index: patchlevel.h > =================================================================== > RCS file: /cvsroot/python/python/dist/src/Include/patchlevel.h,v > retrieving revision 2.56 > retrieving revision 2.57 > diff -C2 -d -r2.56 -r2.57 > *** patchlevel.h 2001/10/19 17:11:58 2.56 > --- patchlevel.h 2001/10/23 22:20:58 2.57 > *************** > *** 27,31 **** > > /* Version as a string */ > ! #define PY_VERSION "2.2b1+" > > /* Version as a single 4-byte hex number, e.g. 0x010502B2 == 1.5.2b2. > --- 27,31 ---- > > /* Version as a string */ > ! #define PY_VERSION "2.2b1" > > /* Version as a single 4-byte hex number, e.g. 0x010502B2 == 1.5.2b2. > > > _______________________________________________ > Python-checkins mailing list > Python-checkins@python.org > http://mail.python.org/mailman/listinfo/python-checkins From guido@python.org Wed Oct 24 02:23:42 2001 From: guido@python.org (Guido van Rossum) Date: Tue, 23 Oct 2001 21:23:42 -0400 Subject: [Python-Dev] Fixing send() Message-ID: <200110240123.VAA06720@cj20424-a.reston1.va.home.com> I've got a possible patch for the send() problem. I'd like review of the patch and the related issues before I apply it. I'd also like more guidance regarding the suitability of this patch for the 2.1.2 bugfix release. (It fixes a real bug, but can sometime lead to different behavior, and adds a new interpretation of the absence of an optional argument, which can be considered a feature.) http://sf.net/tracker/?group_id=5470&atid=305470&func=detail&aid=474307 --Guido van Rossum (home page: http://www.python.org/~guido/) From fdrake@acm.org Wed Oct 24 02:30:27 2001 From: fdrake@acm.org (Fred L. Drake, Jr.) Date: Tue, 23 Oct 2001 21:30:27 -0400 Subject: [Python-Dev] Dictionary subclasses and exec In-Reply-To: <20011023181737.B27568@mems-exchange.org> References: <20011023201329.27143.qmail@mozart.cnri.reston.va.us> <200110232108.f9NL8Fi25565@odiug.zope.com> <20011023181737.B27568@mems-exchange.org> Message-ID: <15318.6579.803083.815943@grendel.zope.com> akuchlin@mems-exchange.org writes: > Oh, well; I need to come up with a different example for subclassing a > built-in type, then. Thanks! <puts on thinking cap> Andrew, You can take a look at that implementation of xml.dom.minidom.NodeList; for Python 2.2, the implementation goes like this: ---------------------------------------------------------------------- class NodeList(list): def item(self, index): if 0 <= index < len(self): return self[index] length = property(lambda self: len(self), doc="The number of nodes in the NodeList.") ---------------------------------------------------------------------- -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> PythonLabs at Zope Corporation From akuchlin@mems-exchange.org Wed Oct 24 02:42:42 2001 From: akuchlin@mems-exchange.org (akuchlin@mems-exchange.org) Date: Tue, 23 Oct 2001 21:42:42 -0400 Subject: [Python-Dev] Dictionary subclasses and exec In-Reply-To: <15318.6579.803083.815943@grendel.zope.com>; from fdrake@acm.org on Tue, Oct 23, 2001 at 09:30:27PM -0400 References: <20011023201329.27143.qmail@mozart.cnri.reston.va.us> <200110232108.f9NL8Fi25565@odiug.zope.com> <20011023181737.B27568@mems-exchange.org> <15318.6579.803083.815943@grendel.zope.com> Message-ID: <20011023214242.A28096@mems-exchange.org> On Tue, Oct 23, 2001 at 09:30:27PM -0400, Fred L. Drake, Jr. wrote: > You can take a look at that implementation of >xml.dom.minidom.NodeList; for Python 2.2, the implementation goes like >this: <vanishingly short implementation deleted> Thanks; maybe something like that will do, and it sparks a stray thought: would it be worth making the file wrapper class in posixfile.py a subclass of file? --amk From fdrake@acm.org Wed Oct 24 02:57:51 2001 From: fdrake@acm.org (Fred L. Drake, Jr.) Date: Tue, 23 Oct 2001 21:57:51 -0400 Subject: [Python-Dev] Dictionary subclasses and exec In-Reply-To: <20011023214242.A28096@mems-exchange.org> References: <20011023201329.27143.qmail@mozart.cnri.reston.va.us> <200110232108.f9NL8Fi25565@odiug.zope.com> <20011023181737.B27568@mems-exchange.org> <15318.6579.803083.815943@grendel.zope.com> <20011023214242.A28096@mems-exchange.org> Message-ID: <15318.8223.646037.669260@grendel.zope.com> akuchlin@mems-exchange.org writes: > Thanks; maybe something like that will do, and it sparks a stray > thought: would it be worth making the file wrapper class in posixfile.py > a subclass of file? Given that the documentation for posixfile delares it to be (essentially) obsolete, I thnk the answer is "no". Code that is obsolete should not be touched as long as it continues to pass any relevant tests and there are no reported bugs. -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> PythonLabs at Zope Corporation From Anthony Baxter <anthony@interlink.com.au> Wed Oct 24 03:23:30 2001 From: Anthony Baxter <anthony@interlink.com.au> (Anthony Baxter) Date: Wed, 24 Oct 2001 12:23:30 +1000 Subject: [Python-Dev] Fixing send() In-Reply-To: Message from Guido van Rossum <guido@python.org> of "Tue, 23 Oct 2001 21:23:42 -0400." <200110240123.VAA06720@cj20424-a.reston1.va.home.com> Message-ID: <200110240223.f9O2NUg29502@mbuna.arbhome.com.au> >>> Guido van Rossum wrote > I've got a possible patch for the send() problem. I'd like review of > the patch and the related issues before I apply it. I'd also like > more guidance regarding the suitability of this patch for the 2.1.2 > bugfix release. (It fixes a real bug, but can sometime lead to > different behavior, and adds a new interpretation of the absence of an > optional argument, which can be considered a feature.) It's a tough decision, because this really _is_ the cleaner solution, but it does stand a slight chance of breaking code, and it's also something that can be worked around in the std library. At the moment I'm planning to just fix the std library modules instead - as these are definitely broken. The added functionality of python's send() is very useful, but it's a feature, not a bug :) Anthony -- Anthony Baxter <anthony@interlink.com.au> It's never too late to have a happy childhood. From jack@oratrix.nl Wed Oct 24 10:25:46 2001 From: jack@oratrix.nl (Jack Jansen) Date: Wed, 24 Oct 2001 11:25:46 +0200 Subject: [Python-Dev] Re: patchlevel.h In-Reply-To: Message by Guido van Rossum <guido@python.org> , Tue, 23 Oct 2001 20:00:44 -0400 , <200110240000.UAA06439@cj20424-a.reston1.va.home.com> Message-ID: <20011024092547.54106303181@snelboot.oratrix.nl> > Jack, you can't do this on the trunk. If you *have* to have a > different patch level, please use a branch! Well, apparently I _can_, I just did. Maybe I _shouldn't_.... :-) I undid it this morning, this mod wasn't supposed to escape from my machine. Maybe I should start using the release branch for Mac releases after you are done with it for the Unix/Windows release. As there is an r22b1 tag to point to the exact revision used for the release there should be no adverse consequences. What do you think? -- Jack Jansen | ++++ stop the execution of Mumia Abu-Jamal ++++ Jack.Jansen@oratrix.com | ++++ if you agree copy these lines to your sig ++++ www.cwi.nl/~jack | see http://www.xs4all.nl/~tank/spg-l/sigaction.htm From fdrake@acm.org Wed Oct 24 12:02:10 2001 From: fdrake@acm.org (Fred L. Drake, Jr.) Date: Wed, 24 Oct 2001 07:02:10 -0400 Subject: [Python-Dev] Re: patchlevel.h In-Reply-To: <20011024092547.54106303181@snelboot.oratrix.nl> References: <guido@python.org> <200110240000.UAA06439@cj20424-a.reston1.va.home.com> <20011024092547.54106303181@snelboot.oratrix.nl> Message-ID: <15318.40882.444530.755092@grendel.zope.com> Jack Jansen writes: > Maybe I should start using the release branch for Mac releases > after you are done with it for the Unix/Windows release. As there > is an r22b1 tag to point to the exact revision used for the release > there should be no adverse consequences. What do you think? That sounds like the right thing to me. -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> PythonLabs at Zope Corporation From guido@python.org Wed Oct 24 13:50:08 2001 From: guido@python.org (Guido van Rossum) Date: Wed, 24 Oct 2001 08:50:08 -0400 Subject: [Python-Dev] Re: patchlevel.h In-Reply-To: Your message of "Wed, 24 Oct 2001 11:25:46 +0200." <20011024092547.54106303181@snelboot.oratrix.nl> References: <20011024092547.54106303181@snelboot.oratrix.nl> Message-ID: <200110241250.IAA09678@cj20424-a.reston1.va.home.com> > Well, apparently I _can_, I just did. Maybe I _shouldn't_.... :-) > > I undid it this morning, this mod wasn't supposed to escape from my machine. Thanks. :-) > Maybe I should start using the release branch for Mac releases after you are > done with it for the Unix/Windows release. As there is an r22b1 tag to point > to the exact revision used for the release there should be no adverse > consequences. What do you think? Fine with me. For you, it might mean more merging of changes though (unless you start working on the Mac port a few days *before* the next beta release :-). --Guido van Rossum (home page: http://www.python.org/~guido/) From Anthony Baxter <anthony@interlink.com.au> Wed Oct 24 14:09:59 2001 From: Anthony Baxter <anthony@interlink.com.au> (Anthony Baxter) Date: Wed, 24 Oct 2001 23:09:59 +1000 Subject: [Python-Dev] "Leaving Sourceforge" article on Advogato In-Reply-To: Message from akuchlin@mems-exchange.org of "Tue, 23 Oct 2001 09:11:26 -0400." <20011023091126.A24293@mems-exchange.org> Message-ID: <200110241309.f9OD9xv32763@mbuna.arbhome.com.au> It's worth re-visiting the advogato page again - there's a post from Pat McGovern at SF there now. >>> akuchlin@mems-exchange.org wrote > There's a discussion of SourceForge alternatives at > http://www.advogato.org/article/357.html, and in a comment someone > mentions that there's an XML export of the data in the tracker. > Unfortunately I can't get it to work following his instructions, but > perhaps someone else can experiment with it more than I can. > > --amk > > _______________________________________________ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > -- Anthony Baxter <anthony@interlink.com.au> It's never too late to have a happy childhood. From barry@zope.com Wed Oct 24 15:10:56 2001 From: barry@zope.com (Barry A. Warsaw) Date: Wed, 24 Oct 2001 10:10:56 -0400 Subject: [Python-Dev] Re: patchlevel.h References: <guido@python.org> <200110240000.UAA06439@cj20424-a.reston1.va.home.com> <20011024092547.54106303181@snelboot.oratrix.nl> Message-ID: <15318.52208.962415.759362@anthem.wooz.org> >>>>> "JJ" == Jack Jansen <jack@oratrix.nl> writes: JJ> I undid it this morning, this mod wasn't supposed to escape JJ> from my machine. Maybe I should start using the release JJ> branch for Mac releases after you are done with it for the JJ> Unix/Windows release. As there is an r22b1 tag to point to the JJ> exact revision used for the release there should be no adverse JJ> consequences. What do you think? Should be fine, although I'd like to add something about it to PEP 101, just to make it official. How about in the "What Next?" section, something like: "Turn ownership of the branch tag over to Jack so he can make the Mac release. He'll typically twiddle with patchlevel.h and may make a few other checkins as needed for the Mac. When he's done, he'll add another tag to the branch: X.YaZ-mac. " That way we'll have the source tarball tag in the branch and the mac release in the branch and we could recreate either. -Barry From jack@oratrix.nl Wed Oct 24 15:16:33 2001 From: jack@oratrix.nl (Jack Jansen) Date: Wed, 24 Oct 2001 16:16:33 +0200 Subject: [Python-Dev] Re: patchlevel.h In-Reply-To: Message by barry@zope.com (Barry A. Warsaw) , Wed, 24 Oct 2001 10:10:56 -0400 , <15318.52208.962415.759362@anthem.wooz.org> Message-ID: <20011024141633.96B6F303181@snelboot.oratrix.nl> > Should be fine, although I'd like to add something about it to PEP > 101, just to make it official. How about in the "What Next?" section, > something like: > > "Turn ownership of the branch tag over to Jack so he can make the Mac > release. He'll typically twiddle with patchlevel.h and may make a few > other checkins as needed for the Mac. > > When he's done, he'll add another tag to the branch: X.YaZ-mac. > " Fine with me, except for the middle sentence, which should go: one of the reasons to go on the branch is that I *don't* have to twiddle patchlevel.h:-) -- Jack Jansen | ++++ stop the execution of Mumia Abu-Jamal ++++ Jack.Jansen@oratrix.com | ++++ if you agree copy these lines to your sig ++++ www.cwi.nl/~jack | see http://www.xs4all.nl/~tank/spg-l/sigaction.htm From barry@zope.com Wed Oct 24 15:22:52 2001 From: barry@zope.com (Barry A. Warsaw) Date: Wed, 24 Oct 2001 10:22:52 -0400 Subject: [Python-Dev] Re: patchlevel.h References: <15318.52208.962415.759362@anthem.wooz.org> <20011024141633.96B6F303181@snelboot.oratrix.nl> Message-ID: <15318.52924.633767.193832@anthem.wooz.org> >>>>> "JJ" == Jack Jansen <jack@oratrix.nl> writes: JJ> Fine with me, except for the middle sentence, which should go: JJ> one of the reasons to go on the branch is that I *don't* have JJ> to twiddle patchlevel.h:-) D'oh! Of course. ;) From martin@v.loewis.de Wed Oct 24 19:18:32 2001 From: martin@v.loewis.de (Martin v. Loewis) Date: Wed, 24 Oct 2001 20:18:32 +0200 Subject: [Python-Dev] Re: patchlevel.h Message-ID: <200110241818.f9OIIWp01350@mira.informatik.hu-berlin.de> > Maybe I should start using the release branch for Mac releases after you are > done with it for the Unix/Windows release. As there is an r22b1 tag to point > to the exact revision used for the release there should be no adverse > consequences. What do you think? As before, I think there should be only a single 2.2b1 release. If you need to provide binaries, use what was released in the 2.2b1 source distribution (which is *not* the Unix release, it is the source distribution). What do your users lose if they have to get 2.2b1 as-is (ie. without further patches)? It may have bugs, sure - but so does the 2.2b1 release for everybody else. There will be a 2.2b2 release, so whatever you commit now will appear in 2.2b2. I think these release schedules are there for a purpose. Regards, Martin From guido@python.org Wed Oct 24 20:13:44 2001 From: guido@python.org (Guido van Rossum) Date: Wed, 24 Oct 2001 15:13:44 -0400 Subject: [Python-Dev] Fixing send() In-Reply-To: Your message of "Wed, 24 Oct 2001 12:23:30 +1000." <200110240223.f9O2NUg29502@mbuna.arbhome.com.au> References: <200110240223.f9O2NUg29502@mbuna.arbhome.com.au> Message-ID: <200110241913.f9OJDiM10434@odiug.zope.com> > >>> Guido van Rossum wrote > > I've got a possible patch for the send() problem. I'd like review of > > the patch and the related issues before I apply it. I'd also like > > more guidance regarding the suitability of this patch for the 2.1.2 > > bugfix release. (It fixes a real bug, but can sometime lead to > > different behavior, and adds a new interpretation of the absence of an > > optional argument, which can be considered a feature.) > It's a tough decision, because this really _is_ the cleaner solution, > but it does stand a slight chance of breaking code, and it's also something > that can be worked around in the std library. At the moment I'm planning to > just fix the std library modules instead - as these are definitely broken. > The added functionality of python's send() is very useful, but it's a > feature, not a bug :) > > Anthony After ample discussion, the PythonLabs crew didn't think my patch would be safe enough -- see the SF tracker entry for a full discussion. Instead, we decided to fix the standard library to always check the return value of send(). Can you help? (If you check in a fix to the 2.1 maintenance branch, you can mark it as "propagate to trunk please".) --Guido van Rossum (home page: http://www.python.org/~guido/) From fredrik@pythonware.com Wed Oct 24 22:52:51 2001 From: fredrik@pythonware.com (Fredrik Lundh) Date: Wed, 24 Oct 2001 23:52:51 +0200 Subject: [Python-Dev] iterator support for SRE? Message-ID: <014801c15cd6$3d18b270$ced241d5@hagrid> a common wish for SRE is a variant of findall that returns all match objects, instead of the strings. recently, I realized that the internal scanner type (originally added to speed up Python versions of findall/sub/split) can be wrapped in an iterator, allowing the user to loop over all matches. >>> import re >>> p = re.compile(somepattern) >>> for match in iter(p.scanner(somestring).search, None): >>> print match.groups() how about adding a "finditer" method, which takes care of the low-level setup: >>> for match in p.finditer(somestring): >>> print match.groups() or should we just make the scanner factory an official part of the SRE interface? </F> From guido@python.org Thu Oct 25 04:35:56 2001 From: guido@python.org (Guido van Rossum) Date: Wed, 24 Oct 2001 23:35:56 -0400 Subject: [Python-Dev] iterator support for SRE? In-Reply-To: Your message of "Wed, 24 Oct 2001 23:52:51 +0200." <014801c15cd6$3d18b270$ced241d5@hagrid> References: <014801c15cd6$3d18b270$ced241d5@hagrid> Message-ID: <200110250335.XAA10393@cj20424-a.reston1.va.home.com> > a common wish for SRE is a variant of findall that returns > all match objects, instead of the strings. > > recently, I realized that the internal scanner type (originally > added to speed up Python versions of findall/sub/split) can > be wrapped in an iterator, allowing the user to loop over all > matches. > > >>> import re > >>> p = re.compile(somepattern) > >>> for match in iter(p.scanner(somestring).search, None): > >>> print match.groups() > > how about adding a "finditer" method, which takes care of > the low-level setup: > > >>> for match in p.finditer(somestring): > >>> print match.groups() > > or should we just make the scanner factory an official part > of the SRE interface? Or both? The scanner interface seems vaguely useful as a low-level tool to create other higher-level variants; but the finditer() call seems to nicely capture a common case. --Guido van Rossum (home page: http://www.python.org/~guido/) From Anthony Baxter <anthony@interlink.com.au> Thu Oct 25 07:24:20 2001 From: Anthony Baxter <anthony@interlink.com.au> (Anthony Baxter) Date: Thu, 25 Oct 2001 16:24:20 +1000 Subject: [Python-Dev] Fixing send() In-Reply-To: Message from Guido van Rossum <guido@python.org> of "Wed, 24 Oct 2001 15:13:44 -0400." <200110241913.f9OJDiM10434@odiug.zope.com> Message-ID: <200110250624.f9P6OKE11623@mbuna.arbhome.com.au> >>> Guido van Rossum wrote > Instead, we decided to fix the standard library to always check the > return value of send(). Can you help? (If you check in a fix to the > 2.1 maintenance branch, you can mark it as "propagate to trunk please".) Will do - I plan to check in a bunch of fixes tomorrow morning. (continuing gdb hell today). One to consider is the status of 'socket.py' - patched, or not patched...? -- Anthony Baxter <anthony@interlink.com.au> It's never too late to have a happy childhood. From sjoerd.mullender@oratrix.com Thu Oct 25 11:36:57 2001 From: sjoerd.mullender@oratrix.com (Sjoerd Mullender) Date: Thu, 25 Oct 2001 12:36:57 +0200 Subject: [Python-Dev] suggestion for PEP 101 Message-ID: <20011025103658.82DCD3021E6@bireme.oratrix.nl> I noticed that the first step in creating a release is ___ Do a CVS update with the -A flag, e.g. % cvs update -A I would suggest adding the -d and -P flags to this command (so cvs update -APd). The -d flag says to create directories that are in the repository but not in the checked-out version (handy for new directories), and the -P flag says to delete (prune) empty directories from the checked-out version (handy for directories that have become empty because all files were deleted). -- Sjoerd Mullender <sjoerd.mullender@oratrix.com> From guido@python.org Thu Oct 25 15:59:34 2001 From: guido@python.org (Guido van Rossum) Date: Thu, 25 Oct 2001 10:59:34 -0400 Subject: [Python-Dev] Fixing send() In-Reply-To: Your message of "Thu, 25 Oct 2001 16:24:20 +1000." <200110250624.f9P6OKE11623@mbuna.arbhome.com.au> References: <200110250624.f9P6OKE11623@mbuna.arbhome.com.au> Message-ID: <200110251459.f9PExZg21822@odiug.zope.com> I'm planning to add a sendall() method to the socket object. See the SF patch: http://sf.net/tracker/index.php?func=detail&aid=474307&group_id=5470&atid=305470 If you have an objection, please let me know. I think this is important enough to bypass the feature freeze -- it's the best way to fix the very real bug (on FreeBSD) of unchecked send() calls. (The alternative would be to add a loop to every use of send().) The sendall() call allows us to fix the standard library modules that currently don't check for send()'s return value, without breaking any existing code, by changing sock.send(data) into sock.sendall(data). Other alternatives intended to fix the problem of unchecked send() calls in standard library module without having to change all send() calls were deemed to dangerous (see the SF patch discussion). I think Anthony is going to add this to the 2.1.2 bugfix release as well -- with the same argument for breaking the feature freeze. --Guido van Rossum (home page: http://www.python.org/~guido/) From Anthony Baxter <anthony@interlink.com.au> Thu Oct 25 16:14:01 2001 From: Anthony Baxter <anthony@interlink.com.au> (Anthony Baxter) Date: Fri, 26 Oct 2001 01:14:01 +1000 Subject: [Python-Dev] Fixing send() In-Reply-To: Message from Guido van Rossum <guido@python.org> of "Thu, 25 Oct 2001 10:59:34 -0400." <200110251459.f9PExZg21822@odiug.zope.com> Message-ID: <200110251514.f9PFE1229217@mbuna.arbhome.com.au> >>> Guido van Rossum wrote > I think Anthony is going to add this to the 2.1.2 bugfix release as > well -- with the same argument for breaking the feature freeze. That's correct. I think it's the appropriate way forward. I'm pretty sure the "goals for patch selection" had an implicit "except where footnote 1 applies" in there :) I'd be curious to know if anyone disagrees violently with this... Anthony -- Anthony Baxter <anthony@interlink.com.au> It's never too late to have a happy childhood. From guido@python.org Thu Oct 25 16:16:29 2001 From: guido@python.org (Guido van Rossum) Date: Thu, 25 Oct 2001 11:16:29 -0400 Subject: [Python-Dev] Fixing send() In-Reply-To: Your message of "Thu, 25 Oct 2001 11:12:23 EDT." <15320.11223.688433.641613@slothrop.digicool.com> References: <200110250624.f9P6OKE11623@mbuna.arbhome.com.au> <200110251459.f9PExZg21822@odiug.zope.com> <15320.11223.688433.641613@slothrop.digicool.com> Message-ID: <200110251516.f9PFGTf24686@odiug.zope.com> [Jeremy] > Can you offer a brief explanation of why we're doing this after all? > I thought we concluded yesterday that we would fix this library. > (That's what the last message before this one said.) Fixing the library would mean adding a loop like this to each application that uses send and currently doesn't check the return value: < sock.send(data) --- > while data: > n = sock.send(data) > data = data[n:] I find this more attractive: < sock.send(data) --- > sock.sendall(data) --Guido van Rossum (home page: http://www.python.org/~guido/) From gmcm@hypernet.com Thu Oct 25 16:36:49 2001 From: gmcm@hypernet.com (Gordon McMillan) Date: Thu, 25 Oct 2001 11:36:49 -0400 Subject: [Python-Dev] Fixing send() In-Reply-To: <200110251459.f9PExZg21822@odiug.zope.com> References: Your message of "Thu, 25 Oct 2001 16:24:20 +1000." <200110250624.f9P6OKE11623@mbuna.arbhome.com.au> Message-ID: <3BD7F951.26534.24848D4A@localhost> [Guido] > I'm planning to add a sendall() method to the socket object. See > the SF patch: > > http://sf.net/tracker/index.php?func=detail&aid=474307&group_id=5 > 470&atid=305470 > > If you have an objection, please let me know. I think this is > important enough to bypass the feature freeze The bonehead who uses sendall with a non-blocking socket will, of course, get an EWOULDBLOCK (or platform equivalent) on an oversize send. Maybe EWOULDBLOCK should get turned into ButtHeadProgrammerError("Don't use sendall with a non-blocking socket")? [I agree that this is the best way to fix the std lib, and that it is very important that it be fixed.] - Gordon From guido@python.org Thu Oct 25 16:42:45 2001 From: guido@python.org (Guido van Rossum) Date: Thu, 25 Oct 2001 11:42:45 -0400 Subject: [Python-Dev] Fixing send() In-Reply-To: Your message of "Thu, 25 Oct 2001 11:36:49 EDT." <3BD7F951.26534.24848D4A@localhost> References: Your message of "Thu, 25 Oct 2001 16:24:20 +1000." <200110250624.f9P6OKE11623@mbuna.arbhome.com.au> <3BD7F951.26534.24848D4A@localhost> Message-ID: <200110251542.f9PFgjF28022@odiug.zope.com> > The bonehead who uses sendall with a non-blocking socket > will, of course, get an EWOULDBLOCK (or platform > equivalent) on an oversize send. Maybe EWOULDBLOCK > should get turned into ButtHeadProgrammerError("Don't use > sendall with a non-blocking socket")? I think natural selection will take care of this breed. :-) > [I agree that this is the best way to fix the std lib, and that it is > very important that it be fixed.] Thanks! --Guido van Rossum (home page: http://www.python.org/~guido/) From barry@zope.com Thu Oct 25 16:45:02 2001 From: barry@zope.com (Barry A. Warsaw) Date: Thu, 25 Oct 2001 11:45:02 -0400 Subject: [Python-Dev] suggestion for PEP 101 References: <20011025103658.82DCD3021E6@bireme.oratrix.nl> Message-ID: <15320.13182.670145.233045@anthem.wooz.org> >>>>> "SM" == Sjoerd Mullender <sjoerd.mullender@oratrix.com> writes: SM> I noticed that the first step in creating a release is | ___ Do a CVS update with the -A flag, e.g. | % cvs update -A SM> I would suggest adding the -d and -P flags to this command (so SM> cvs update -APd). Good point, thanks. -Barry From fdrake@acm.org Thu Oct 25 16:56:18 2001 From: fdrake@acm.org (Fred L. Drake) Date: Thu, 25 Oct 2001 11:56:18 -0400 (EDT) Subject: [Python-Dev] [development doc updates] Message-ID: <20011025155618.C21CA28697@cj42289-a.reston1.va.home.com> The development version of the documentation has been updated: http://python.sourceforge.net/devel-docs/ Experimental stylesheet changes -- code is now presented using proportional fonts (mostly since Guido really dislikes Courier). Please send comments on this change to me at python-docs@python.org. From jeremy@zope.com Thu Oct 25 17:18:40 2001 From: jeremy@zope.com (Jeremy Hylton) Date: Thu, 25 Oct 2001 12:18:40 -0400 (EDT) Subject: [Python-Dev] Fixing send() In-Reply-To: <3BD7F951.26534.24848D4A@localhost> References: <200110250624.f9P6OKE11623@mbuna.arbhome.com.au> <3BD7F951.26534.24848D4A@localhost> Message-ID: <15320.15200.184832.18268@slothrop.digicool.com> If we're making the std library routines that use sockets robust, I expect we ought to think about exceptions that should be dealt with. My Linux man page suggests that ENOBUFS and EINTR are errors that indicate "try again." There are probably other such errors on other platforms. We use Python sockets and asyncore in ZEO (a Zope subsystem) and I've found that every weird error that I've never heard of seems to occur in the wild. If we don't deal with these errors, then we've haven't fully succeeded in making the calls robust. Jeremy From guido@python.org Thu Oct 25 19:03:13 2001 From: guido@python.org (Guido van Rossum) Date: Thu, 25 Oct 2001 14:03:13 -0400 Subject: [Python-Dev] Fixing send() In-Reply-To: Your message of "Thu, 25 Oct 2001 12:18:40 EDT." <15320.15200.184832.18268@slothrop.digicool.com> References: <200110250624.f9P6OKE11623@mbuna.arbhome.com.au> <3BD7F951.26534.24848D4A@localhost> <15320.15200.184832.18268@slothrop.digicool.com> Message-ID: <200110251803.f9PI3D428392@odiug.zope.com> > If we're making the std library routines that use sockets robust, I > expect we ought to think about exceptions that should be dealt with. > My Linux man page suggests that ENOBUFS and EINTR are errors that > indicate "try again." There are probably other such errors on other > platforms. > > We use Python sockets and asyncore in ZEO (a Zope subsystem) and I've > found that every weird error that I've never heard of seems to occur > in the wild. If we don't deal with these errors, then we've haven't > fully succeeded in making the calls robust. There's a difference though. The bug I'm trying to fix with sendall() was a silent failure, causing at best protocol failures later, at worst silently lost data. ENOBUFS or EINTR etc. cause clear exceptions when they happen. --Guido van Rossum (home page: http://www.python.org/~guido/) From jeremy@zope.com Thu Oct 25 22:15:39 2001 From: jeremy@zope.com (Jeremy Hylton) Date: Thu, 25 Oct 2001 17:15:39 -0400 (EDT) Subject: [Python-Dev] dict comps Message-ID: <15320.33019.723537.580070@slothrop.digicool.com> We agreed yesterday that the dictionary() constructor would accept a a list of two-tuples (strictly speaking an iterable object of iterable objects of length 2). That plus list comprehensions pretty much covers the territory of dict comprehensions: >>> print dictionary([(i, chr(65 + i)) for i in range(4)]) {0: 'A', 1: 'B', 2: 'C', 3: 'D'} Jeremy From andymac@bullseye.apana.org.au Thu Oct 25 14:15:24 2001 From: andymac@bullseye.apana.org.au (Andrew MacIntyre) Date: Thu, 25 Oct 2001 23:15:24 +1000 (EST) Subject: [Python-Dev] status of OS/2 ports of Python In-Reply-To: <323.1003777312.538787.209802503@lilbastard> Message-ID: <Pine.OS2.4.32.0110252249260.577-100000@central> Unfortunately I haven't had the bandwidth to vigorously pursue completing integration of my previously submitted patches, and at this time don't expect to be able to do much until about the time 2.2b2 is scheduled to hit the streets :-(. So I don't expect that 2.2 will have OS/2+EMX build support in the sourceball... At this stage, my plan is to finalise a binary release of 2.2 before submitting a new set of patches, taking into account Martin von Loewis' comments and suggestions (much appreciated thank you!) among others. I also plan to try and coordinate my patches with those of Michael Muller (473749, 474169 & 474500 so far) for the VAC++ build of Python for OS/2. I will try and review Michael's patches with a view to getting them into 2.2, as the VAC++ build has been part of the sourceball for a few versions. -- Andrew I MacIntyre "These thoughts are mine alone..." E-mail: andymac@bullseye.apana.org.au | Snail: PO Box 370 andymac@pcug.org.au | Belconnen ACT 2616 Web: http://www.andymac.org/ | Australia From guido@python.org Thu Oct 25 22:30:22 2001 From: guido@python.org (Guido van Rossum) Date: Thu, 25 Oct 2001 17:30:22 -0400 Subject: [Python-Dev] status of OS/2 ports of Python In-Reply-To: Your message of "Thu, 25 Oct 2001 23:15:24 +1000." <Pine.OS2.4.32.0110252249260.577-100000@central> References: <Pine.OS2.4.32.0110252249260.577-100000@central> Message-ID: <200110252130.f9PLUM329494@odiug.zope.com> > Unfortunately I haven't had the bandwidth to vigorously pursue completing > integration of my previously submitted patches, and at this time don't > expect to be able to do much until about the time 2.2b2 is scheduled to > hit the streets :-(. So I don't expect that 2.2 will have OS/2+EMX build > support in the sourceball... > > At this stage, my plan is to finalise a binary release of 2.2 before > submitting a new set of patches, taking into account Martin von Loewis' > comments and suggestions (much appreciated thank you!) among others. I > also plan to try and coordinate my patches with those of Michael Muller > (473749, 474169 & 474500 so far) for the VAC++ build of Python for OS/2. > > I will try and review Michael's patches with a view to getting them into > 2.2, as the VAC++ build has been part of the sourceball for a few > versions. Andy, would it help if you had SourceForge commit privileges? You'd be obliged to only check in code that doesn't break the build on other platforms, so you would be required to test anything you want to check in on another platform (Linux or Windows) before you commit to save you the embarrassment of breaking the build, but if (as I expect) you would mostly be adding stuff inside "#ifdef OS2", that shouldn't be a big burden. It would save you the trouble of uploading patches to SourceForge and it would save *us* the trouble of reviewing your patches and checking them in. All your code would be owned by the PSF, but that's what you want anyway, right? --Guido van Rossum (home page: http://www.python.org/~guido/) From skip@pobox.com (Skip Montanaro) Thu Oct 25 22:53:27 2001 From: skip@pobox.com (Skip Montanaro) (Skip Montanaro) Date: Thu, 25 Oct 2001 16:53:27 -0500 Subject: [Python-Dev] weird gdb/python/gtk/pango interaction? Message-ID: <15320.35287.392870.291275@beluga.mojam.com> Folks, I'm no longer able to run PyGtk scripts from gdb under some circumstances. I am using the CVS HEAD of glib, atk, pango, gtk+ and pygtk. The following simple Python script works fine when run from gdb: #!/usr/bin/env python import gtk w = gtk.Window() w.connect("destroy", gtk.mainquit) #lbl = gtk.Label("hi") #w.add(lbl) w.show_all() gtk.mainloop() with the following output: (gdb) r basic.py Starting program: /usr/local/bin/python basic.py [New Thread 1024 (LWP 19044)] Gtk-Message: YOU ARE USING THE DEVEL BRANCH 1.3.x OF GTK+ WHICH IS CURRENTLY UNDER HEAVY DEVELOPMENT AND FREQUENTLY INTRODUCES INSTABILITIES. if you don't know why you are getting this, you probably want to use the stable branch which can be retrieved from ftp://ftp.gtk.org/pub/gtk/v1.2/ or via CVS with cvs checkout -r glib-1-2 glib; cvs checkout -r gtk-1-2 gtk+ Program exited normally. If I uncomment the two lines involving lbl, however, I get this mess: (gdb) r basic.py Starting program: /usr/local/bin/python basic.py [New Thread 1024 (LWP 19127)] Gtk-Message: YOU ARE USING THE DEVEL BRANCH 1.3.x OF GTK+ WHICH IS CURRENTLY UNDER HEAVY DEVELOPMENT AND FREQUENTLY INTRODUCES INSTABILITIES. if you don't know why you are getting this, you probably want to use the stable branch which can be retrieved from ftp://ftp.gtk.org/pub/gtk/v1.2/ or via CVS with cvs checkout -r glib-1-2 glib; cvs checkout -r gtk-1-2 gtk+ basic.py (pid:19127): GRuntime-CRITICAL **: file gparamspecs.c: line 1687 (g_param_spec_float): assertion `default_value >= minimum && default_value <= maximum' failed basic.py (pid:19127): GRuntime-CRITICAL **: file gobject.c: line 270 (g_object_class_install_property): assertion `G_IS_PARAM_SPEC (pspec)' failed basic.py (pid:19127): ** WARNING **: Couldn't load font "Sans -2.09715e+06" falling back to "Sans -2.09715e+06" basic.py (pid:19127): ** WARNING **: Couldn't load font "Sans -2.09715e+06" falling back to "Sans -2.09715e+06" basic.py (pid:19127): ** WARNING **: All font failbacks failed!!!! Program exited with code 01. If I change the Label to an EventBox, everything works, presumably because no font rendering is involved. Also when I run the script from the bash prompt it works as expected (no Gtk warnings). The equivalent C program: #include <gtk/gtk.h> int main( int argc, char *argv[] ) { GtkWidget *window; GtkWidget *label; gtk_init(&argc, &argv); window = gtk_window_new (GTK_WINDOW_TOPLEVEL); gtk_signal_connect (GTK_OBJECT (window), "destroy", GTK_SIGNAL_FUNC(gtk_main_quit), NULL); gtk_window_set_title (GTK_WINDOW (window), ""); label = gtk_label_new("hi"); gtk_container_add(GTK_CONTAINER(window), label); gtk_widget_show_all (GTK_WIDGET(window)); gtk_main(); return(0); } works as expected, both from the command line and if run from the gdb prompt. This wouldn't be such a big deal, except in a more complex script I get segfaults in some circumstances, and this makes debugging that problem a bit difficult. Any idea what's going on? -- Skip Montanaro (skip@pobox.com) http://www.mojam.com/ http://www.musi-cal.com/ From martin@v.loewis.de Thu Oct 25 23:31:57 2001 From: martin@v.loewis.de (Martin v. Loewis) Date: Fri, 26 Oct 2001 00:31:57 +0200 Subject: [Python-Dev] weird gdb/python/gtk/pango interaction? Message-ID: <200110252231.f9PMVvW03121@mira.informatik.hu-berlin.de> > Any idea what's going on? When you see that a program behaves differently in gdb than it does on the shell, the most likely cause is that gdb starts another shell when running the program, I believe essentially through system("$SHELL -c command") (see gdb documentation and source for details). For example, some of the $HOME/.something files may set PYTHONPATH to some interesting value. If you see that a Python program behaves differently from an "equivalent" C code, well, the most likely cause is that the programs are not equivalent. One obvious (to me) difference is that one calls set_title, when the other doesn't. Regards, Martin From aahz@rahul.net Fri Oct 26 02:12:45 2001 From: aahz@rahul.net (Aahz Maruch) Date: Thu, 25 Oct 2001 18:12:45 -0700 (PDT) Subject: [Python-Dev] Fixing send() In-Reply-To: <200110251514.f9PFE1229217@mbuna.arbhome.com.au> from "Anthony Baxter" at Oct 26, 2001 01:14:01 AM Message-ID: <20011026011245.65372E8C3@waltz.rahul.net> Anthony Baxter wrote: >Guido van Rossum wrote >> >> I think Anthony is going to add this to the 2.1.2 bugfix release as >> well -- with the same argument for breaking the feature freeze. > > That's correct. I think it's the appropriate way forward. > > I'm pretty sure the "goals for patch selection" had an implicit > "except where footnote 1 applies" in there :) PEP 6 has an explicit exception for BDFL Pronouncements. ;-) Given that this is "guaranteed" to not break any code, it's fair game. -- --- Aahz (@pobox.com) Hugs and backrubs -- I break Rule 6 <*> http://www.rahul.net/aahz/ Androgynous poly kinky vanilla queer het Pythonista We must not let the evil of a few trample the freedoms of the many. From fdrake@acm.org Fri Oct 26 04:13:46 2001 From: fdrake@acm.org (Fred L. Drake) Date: Thu, 25 Oct 2001 23:13:46 -0400 (EDT) Subject: [Python-Dev] [development doc updates] Message-ID: <20011026031346.8550628697@beowolf.digicool.com> The development version of the documentation has been updated: http://python.sourceforge.net/devel-docs/ Yet another experimental change to the presentation -- ditch the proportional font for code since some displays (especially interactive sessions and tabular displays) get messed up with proportional fonts. We do try to use monospaced fonts that are less ugly than Courier. As before, feedback on the fonts is welcome at python-docs@python.org. From tim.one@home.com Fri Oct 26 06:34:16 2001 From: tim.one@home.com (Tim Peters) Date: Fri, 26 Oct 2001 01:34:16 -0400 Subject: [Python-Dev] dict comps In-Reply-To: <15320.33019.723537.580070@slothrop.digicool.com> Message-ID: <LNBBLJKPBEHFEDALKOLCOECOMDAA.tim.one@home.com> [Jeremy Hylton] > We agreed yesterday that the dictionary() constructor would accept a > a list of two-tuples (strictly speaking an iterable object of iterable > objects of length 2). FYI, this is checked in now. > That plus list comprehensions pretty much covers the territory of > dict comprehensions: > > >>> print dictionary([(i, chr(65 + i)) for i in range(4)]) > {0: 'A', 1: 'B', 2: 'C', 3: 'D'} Wow -- that's *exactly* what it prints. You got your own time machine now? While it covers the semantics, the pragmatics may be off, since listcomps produce genuine lists, and so e.g. dictionary([(key, f(key)) for key in file('huge')]) may require constructing an unboundedly large list of twoples before dictionary() sees the first pair. dictionary() per se doesn't require materializing a giant list in one gulp. From mal@lemburg.com Fri Oct 26 10:02:02 2001 From: mal@lemburg.com (M.-A. Lemburg) Date: Fri, 26 Oct 2001 11:02:02 +0200 Subject: [Python-Dev] dict comps References: <LNBBLJKPBEHFEDALKOLCOECOMDAA.tim.one@home.com> Message-ID: <3BD9268A.72AD71E9@lemburg.com> Tim Peters wrote: > > [Jeremy Hylton] > > We agreed yesterday that the dictionary() constructor would accept a > > a list of two-tuples (strictly speaking an iterable object of iterable > > objects of length 2). > > FYI, this is checked in now. > > > That plus list comprehensions pretty much covers the territory of > > dict comprehensions: > > > > >>> print dictionary([(i, chr(65 + i)) for i in range(4)]) > > {0: 'A', 1: 'B', 2: 'C', 3: 'D'} Cool ! > Wow -- that's *exactly* what it prints. You got your own time machine now? > > While it covers the semantics, the pragmatics may be off, since listcomps > produce genuine lists, and so e.g. > > dictionary([(key, f(key)) for key in file('huge')]) > > may require constructing an unboundedly large list of twoples before > dictionary() sees the first pair. dictionary() per se doesn't require > materializing a giant list in one gulp. One way or another, you'll use up a giant chunk or two of data on the heap... I'd suggest adding a new builtin huge_file_as_mapping_apply('file', f) ;-) Seriously, this goes down the path of lazy evaluation of expressions. Not sure whether this is the right path to follow though (can cause brain damage due to overloaded builtin debuggers, IMHO). BTW, looks like I can finally get rid off the dict() builtin I have in mxTools which is Good News ! -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Consulting & Company: http://www.egenix.com/ Python Software: http://www.lemburg.com/python/ From mal@lemburg.com Fri Oct 26 09:52:50 2001 From: mal@lemburg.com (M.-A. Lemburg) Date: Fri, 26 Oct 2001 10:52:50 +0200 Subject: [Python-Dev] weird gdb/python/gtk/pango interaction? References: <15320.35287.392870.291275@beluga.mojam.com> Message-ID: <3BD92462.8A8E2C4F@lemburg.com> Skip Montanaro wrote: > > Folks, > > I'm no longer able to run PyGtk scripts from gdb under some circumstances. > I am using the CVS HEAD of glib, atk, pango, gtk+ and pygtk. The following > simple Python script works fine when run from gdb: > > #!/usr/bin/env python > > import gtk > > w = gtk.Window() > w.connect("destroy", gtk.mainquit) > #lbl = gtk.Label("hi") > #w.add(lbl) > > w.show_all() > gtk.mainloop() > > with the following output: > > (gdb) r basic.py > Starting program: /usr/local/bin/python basic.py > [New Thread 1024 (LWP 19044)] > Gtk-Message: YOU ARE USING THE DEVEL BRANCH 1.3.x OF GTK+ WHICH IS CURRENTLY > UNDER HEAVY DEVELOPMENT AND FREQUENTLY INTRODUCES INSTABILITIES. if you don't know why you are getting this, you probably want to use the stable branch which can be retrieved from > ftp://ftp.gtk.org/pub/gtk/v1.2/ or via CVS with > cvs checkout -r glib-1-2 glib; cvs checkout -r gtk-1-2 gtk+ Just curious: does the proposed stable branch show the same behaviour ? If so, I'd suggest to ask the GTK guys for help. If not, then perhaps you have to rebuild the Python extensions involved and make sure that they match the Python version you are using. What you're seeing looks like there are some buffer overruns going on... I've never seen such a strange font name ;-) -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Consulting & Company: http://www.egenix.com/ Python Software: http://www.lemburg.com/python/ From fredrik@pythonware.com Fri Oct 26 10:38:29 2001 From: fredrik@pythonware.com (Fredrik Lundh) Date: Fri, 26 Oct 2001 11:38:29 +0200 Subject: [Python-Dev] iterator support for SRE? References: <014801c15cd6$3d18b270$ced241d5@hagrid> <200110250335.XAA10393@cj20424-a.reston1.va.home.com> Message-ID: <015a01c15e01$f99f4ef0$0900a8c0@spiff> guido wrote: > > how about adding a "finditer" method, which takes care of > > the low-level setup: > > > > >>> for match in p.finditer(somestring): > > >>> print match.groups() > > > > or should we just make the scanner factory an official part > > of the SRE interface? > > Or both? The scanner interface seems vaguely useful as a low-level > tool to create other higher-level variants; but the finditer() call > seems to nicely capture a common case. as you might have noticed, I went ahead and checked in code for "finditer". nobody has complained this far ;-) isn't "iterfind" a better name, btw? more like "iterkeys" (etc). if nobody complains, I'll change the name, and check in some documentation. ::: will have to think about scanner; I agree that it may seem to be vaguely useful, but cannot think of a use case that isn't better handled by findall/split/sub or iterfind. can anyone? </F> From jack@oratrix.nl Fri Oct 26 11:25:39 2001 From: jack@oratrix.nl (Jack Jansen) Date: Fri, 26 Oct 2001 12:25:39 +0200 Subject: [Python-Dev] Re: patchlevel.h In-Reply-To: Message by "Martin v. Loewis" <martin@v.loewis.de> , Wed, 24 Oct 2001 20:18:32 +0200 , <200110241818.f9OIIWp01350@mira.informatik.hu-berlin.de> Message-ID: <20011026102540.3DBEB303181@snelboot.oratrix.nl> > > Maybe I should start using the release branch for Mac releases after you are > > done with it for the Unix/Windows release. As there is an r22b1 tag to point > > to the exact revision used for the release there should be no adverse > > consequences. What do you think? > > As before, I think there should be only a single 2.2b1 release. If you > need to provide binaries, use what was released in the 2.2b1 source > distribution (which is *not* the Unix release, it is the source > distribution). Martin, while I completely agree with you in principle, in practice this isn't possible. When the release branch is made there's usually a flurry of minor fixes, and these are almost guaranteed to break something on the mac. Tiny things, like missing casts, but breaks nonetheless. Moreover, the "release loop" is now about 24 hours long, according to PEP101, and even extending it seriously (like to about a week) still wouldn't guarantee that I could react timely. Not only am I 6 hours away from the unix/win release folks, but I also have a paying job that is less and less MacPython-related, so I can't firmly commit myself. And it has happened to me already (twice, I think) that there was a showstopper bug on the Mac that has caused me to either be very late with a release or skip one altogether. This happens on the Mac more often than on Unix/Windows, because MacPython relies on a third party unix I/O emulation library. -- Jack Jansen | ++++ stop the execution of Mumia Abu-Jamal ++++ Jack.Jansen@oratrix.com | ++++ if you agree copy these lines to your sig ++++ www.cwi.nl/~jack | see http://www.xs4all.nl/~tank/spg-l/sigaction.htm From thomas.heller@ion-tof.com Fri Oct 26 14:29:03 2001 From: thomas.heller@ion-tof.com (Thomas Heller) Date: Fri, 26 Oct 2001 15:29:03 +0200 Subject: [Python-Dev] innocent question ;-) Message-ID: <024501c15e22$2dd753f0$e000a8c0@thomasnotebook> ... about my beloved buffer object: Would a patch to make the buffer object subclassable be still accepted for 2.2, or does this fall under the feature freeze? Another question: It is my understanding that makeing an object subclassable also exposes its constructor to Python. What is the signature of this constructor? Is the three argument version xxx(name, bases, dict) always used to return a new type object (Don Beaudry hook), and can other signatures freely be used to creates objects/instances of this type/class? Thomas From guido@python.org Fri Oct 26 14:30:55 2001 From: guido@python.org (Guido van Rossum) Date: Fri, 26 Oct 2001 09:30:55 -0400 Subject: [Python-Dev] iterator support for SRE? In-Reply-To: Your message of "Fri, 26 Oct 2001 11:38:29 +0200." <015a01c15e01$f99f4ef0$0900a8c0@spiff> References: <014801c15cd6$3d18b270$ced241d5@hagrid> <200110250335.XAA10393@cj20424-a.reston1.va.home.com> <015a01c15e01$f99f4ef0$0900a8c0@spiff> Message-ID: <200110261330.JAA18598@cj20424-a.reston1.va.home.com> > as you might have noticed, I went ahead and checked in code > for "finditer". nobody has complained this far ;-) Far from it. :-) > isn't "iterfind" a better name, btw? more like "iterkeys" (etc). > > if nobody complains, I'll change the name, and check in some > documentation. Hm, but finditer() is more like findall(), and that counts more I'd say. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Fri Oct 26 14:42:53 2001 From: guido@python.org (Guido van Rossum) Date: Fri, 26 Oct 2001 09:42:53 -0400 Subject: [Python-Dev] innocent question ;-) In-Reply-To: Your message of "Fri, 26 Oct 2001 15:29:03 +0200." <024501c15e22$2dd753f0$e000a8c0@thomasnotebook> References: <024501c15e22$2dd753f0$e000a8c0@thomasnotebook> Message-ID: <200110261342.JAA18712@cj20424-a.reston1.va.home.com> > ... about my beloved buffer object: > > Would a patch to make the buffer object subclassable > be still accepted for 2.2, or does this fall under > the feature freeze? NO. Go away. :-) I think the buffer() builtin should be deprecated, not improved. > Another question: > It is my understanding that makeing an object > subclassable also exposes its constructor to Python. Yes, since otherwise there would be no way to instantiate the subclass either. > What is the signature of this constructor? Is the > three argument version xxx(name, bases, dict) always used > to return a new type object (Don Beaudry hook), and can other > signatures freely be used to creates objects/instances > of this type/class? I'm sorry, I don't understand the question. This is the constructor signature for type object, IOW for metaclasses. It has nothing to do with the constructor signature for builtin classes. Also, what do you mean by constructor? __new__ or __init__? Read test_descr.py for examples. Also xxsubtype.c and various built-in types if you're interested in doing this at the C level. --Guido van Rossum (home page: http://www.python.org/~guido/) From thomas.heller@ion-tof.com Fri Oct 26 14:57:23 2001 From: thomas.heller@ion-tof.com (Thomas Heller) Date: Fri, 26 Oct 2001 15:57:23 +0200 Subject: [Python-Dev] innocent question ;-) References: <024501c15e22$2dd753f0$e000a8c0@thomasnotebook> <200110261342.JAA18712@cj20424-a.reston1.va.home.com> Message-ID: <02e901c15e26$2314abd0$e000a8c0@thomasnotebook> From: "Guido van Rossum" <guido@python.org> > > ... about my beloved buffer object: > > > > Would a patch to make the buffer object subclassable > > be still accepted for 2.2, or does this fall under > > the feature freeze? > > NO. Go away. :-) Ok. ;-( Is the list of subtypable types now complete (for 2.2), or do you indend to do more? > > I think the buffer() builtin should be deprecated, not improved. > > > Another question: > > It is my understanding that makeing an object > > subclassable also exposes its constructor to Python. > > Yes, since otherwise there would be no way to instantiate the > subclass either. So I understood this... > > > What is the signature of this constructor? Is the > > three argument version xxx(name, bases, dict) always used > > to return a new type object (Don Beaudry hook), and can other > > signatures freely be used to creates objects/instances > > of this type/class? > > I'm sorry, I don't understand the question. This is the constructor > signature for type object, IOW for metaclasses. It has nothing to do > with the constructor signature for builtin classes. ...but not this, sorry. I've confused this: >>> int(10) 10 with this: >>> type(int)("name", (), {}) <class '__main__.name'> Sorry again. Thomas From paul-python@svensson.org Fri Oct 26 15:01:56 2001 From: paul-python@svensson.org (Paul Svensson) Date: Fri, 26 Oct 2001 10:01:56 -0400 (EDT) Subject: [Python-Dev] dict comps In-Reply-To: <15320.33019.723537.580070@slothrop.digicool.com> Message-ID: <Pine.LNX.4.21.0110260958270.27030-100000@familjen.svensson.org> On Thu, 25 Oct 2001, Jeremy Hylton wrote: >We agreed yesterday that the dictionary() constructor would accept a >a list of two-tuples (strictly speaking an iterable object of iterable >objects of length 2). That plus list comprehensions pretty much >covers the territory of dict comprehensions: > >>>> print dictionary([(i, chr(65 + i)) for i in range(4)]) >{0: 'A', 1: 'B', 2: 'C', 3: 'D'} > Wouldn't it be nice to be able to spell that { i: chr(65 + i) for i in range(4) } /Paul From guido@python.org Fri Oct 26 15:03:25 2001 From: guido@python.org (Guido van Rossum) Date: Fri, 26 Oct 2001 10:03:25 -0400 Subject: [Python-Dev] innocent question ;-) In-Reply-To: Your message of "Fri, 26 Oct 2001 15:57:23 +0200." <02e901c15e26$2314abd0$e000a8c0@thomasnotebook> References: <024501c15e22$2dd753f0$e000a8c0@thomasnotebook> <200110261342.JAA18712@cj20424-a.reston1.va.home.com> <02e901c15e26$2314abd0$e000a8c0@thomasnotebook> Message-ID: <200110261403.KAA18833@cj20424-a.reston1.va.home.com> > Is the list of subtypable types now complete (for 2.2), > or do you indend to do more? I just found a few extension modules that define their own types, like SocketType. I'd like to see those converted. I don't know if I have time. --Guido van Rossum (home page: http://www.python.org/~guido/) From jack@oratrix.nl Fri Oct 26 15:23:31 2001 From: jack@oratrix.nl (Jack Jansen) Date: Fri, 26 Oct 2001 16:23:31 +0200 Subject: [Python-Dev] innocent question ;-) In-Reply-To: Message by Guido van Rossum <guido@python.org> , Fri, 26 Oct 2001 10:03:25 -0400 , <200110261403.KAA18833@cj20424-a.reston1.va.home.com> Message-ID: <20011026142331.C8E84303181@snelboot.oratrix.nl> > > Is the list of subtypable types now complete (for 2.2), > > or do you indend to do more? > > I just found a few extension modules that define their own types, like > SocketType. I'd like to see those converted. I don't know if I have > time. At what point can we consider the API stable? All this inheriting and subtyping seems like it would be absolutely brilliant for the Mac toolbox APIs (and I suspect also for the PythonWin API's, Mark?) but I've refrained from even looking at it (well, I did have a quick peek:-), because (a) I have no time, (b) I don't want to chase a changing API and (c) there's some methodchain-based poor-mans-inheritance in there already that will undoubtedly break. -- Jack Jansen | ++++ stop the execution of Mumia Abu-Jamal ++++ Jack.Jansen@oratrix.com | ++++ if you agree copy these lines to your sig ++++ www.cwi.nl/~jack | see http://www.xs4all.nl/~tank/spg-l/sigaction.htm From webmaster@garymgordon.com Fri Oct 26 15:30:10 2001 From: webmaster@garymgordon.com (Gary M. Gordon) Date: Fri, 26 Oct 2001 09:30:10 -0500 Subject: [Python-Dev] Which one of our FREE website enhancements would you like? Message-ID: <E15x7zu-0005Pa-00@thunder5.cwihosting.com> <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN"> <HTML><HEAD><TITLE>Gary M. Gordon - FREE Website Enhancement Promotional Offer</TITLE> <META http-equiv=Content-Type content="text/html; charset=us-ascii"> </HEAD> <BODY bgcolor="#ffffee"> <hr solid color="#336699" width="475" align="left"> <br> <DIV> <FONT face=Verdana size="4" color="#ff0000"><STRONG>FREE WEBSITE ENHANCEMENT OFFER<br> yes, ... absolutely FREE</strong></font><STRONG> !</STRONG><BR><br></DIV><SPAN class=288043613-22102001></SPAN> <DIV><FONT face=Verdana color=#000040 size=2><SPAN class=288043613-22102001> </SPAN> (No contracts! No commitments! No cost!)</FONT></DIV> <DIV><FONT face=Verdana color=#000040 size=2><SPAN class=288043613-22102001> <EM>- Links to sample enhancements are provided below. -</EM></SPAN></FONT></DIV><FONT face=Arial size=2></FONT><FONT face=Arial size=2></FONT><FONT face=Arial size=2></FONT><FONT face=Arial size=2></FONT> <DIV><FONT face=Arial size=2></FONT><FONT face=Arial size=2></FONT><FONT face=Arial size=2></FONT><FONT face=Arial size=2></FONT><BR><FONT face=Verdana color=#000040 size=2><STRONG>As a way to introduce <SPAN class=288043613-22102001>our</SPAN> company to you, we are going to give</STRONG></FONT></DIV> <DIV><FONT face=Verdana color=#000040 size=2><STRONG>you<SPAN class=288043613-22102001> </SPAN>an enhancement that you can add to your website<SPAN class=288043613-22102001> ............</SPAN></STRONG></FONT></DIV> <DIV><FONT size=+0><STRONG><SPAN class=288043613-22102001></SPAN><FONT face=Verdana><FONT color=#000040><FONT size=2><SPAN class=288043613-22102001> </SPAN> absolutely <FONT color=#ff0000>FREE</FONT>!</FONT></FONT></FONT></STRONG></FONT></DIV> <DIV><FONT size=+0><FONT face=Arial size=2></FONT><FONT face=Arial size=2></FONT><FONT face=Arial size=2></FONT><FONT face=Arial size=2></FONT><FONT face=Arial size=2></FONT><FONT face=Arial size=2></FONT><FONT face=Arial size=2></FONT><FONT face=Arial size=2></FONT><BR><FONT face=Verdana color=#000040 size=2>==========================================<SPAN class=288043613-22102001>===</SPAN>======</FONT></FONT></DIV> <DIV><FONT face=Verdana color=#000040 size=2></FONT> </DIV> <DIV><FONT face=Verdana color=#000040 size=2> Just select from one of our <STRONG><FONT color=#ff0000>FREE</FONT></STRONG> website enhancement features<BR> that are listed below. <STRONG>There are 'No contracts required!'</STRONG> ..<BR> </FONT><BLOCKQUOTE><FONT face=Verdana color=#000040 size=2><STRONG>There are ... </STRONG></font><UL><FONT face=Verdana color=#000040 size=2><LI> <STRONG>No commitments!<BR></strong><STRONG>No costs!</STRONG><BR>and ...<BR><STRONG>we will also set everything up ... for <FONT color=#ff0000>FREE</FONT>!</STRONG></LI></font> </UL></BLOCKQUOTE></DIV> <DIV><FONT face=Verdana color=#000040 size=2> Let me introduce myself ...</FONT></DIV> <DIV><FONT face=Verdana color=#000040 size=2></FONT> </DIV> <DIV><FONT face=Verdana color=#000040 size=2> My name is Gary Gordon. I am the owner of Gary M. Gordon, LLC<BR> a WEB DEVELOPMENT COMPANY which provides complete solutions<BR> for the needs of small to large businesses. (No project is ever to<BR> small or to large.) If you need a <SPAN class=288043613-22102001>'</SPAN>fix<SPAN class=288043613-22102001>'</SPAN> or <SPAN class=288043613-22102001>'</SPAN>enhancement<SPAN class=288043613-22102001>'</SPAN> added to<BR> a current website ... that's <SPAN class=288043613-22102001>perfect</SPAN>. If you need to have a new<BR> website developed from scratch ... that's <SPAN class=288043613-22102001>great</SPAN>. <SPAN class=288043613-22102001> </SPAN><STRONG><EM>We do it all!</EM></STRONG></FONT></DIV> <DIV><FONT face=Verdana color=#000040 size=2></FONT> </DIV> <DIV><FONT face=Verdana color=#000040 size=2></FONT> </DIV> <DIV><FONT color=#000040><FONT face=Verdana><FONT size=2> <STRONG><EM><FONT color=#000000><FONT color=#ff0000>FREE</FONT> WEBSITE <SPAN class=288043613-22102001>ENHANCEMENTS</SPAN> YOU CAN CHOOSE FROM:</FONT></EM></STRONG></FONT></FONT></FONT></DIV> <DIV><FONT face=Verdana color=#000040 size=2></FONT> </DIV> <DIV><FONT face=Verdana color=#000040 size=2> (Simply click on the 'http:// link' provided below the title of<BR> each item to view just a sample of the FREE enhancement.)</FONT></DIV> <DIV><FONT face=Verdana color=#000040 size=2></FONT> </DIV> <DIV><FONT face=Verdana color=#000040 size=2> <STRONG>1)</STRONG> <STRONG><FONT color=#ff0000>FREE 'Macromedia Flash (.SWF FORMAT) Animation'<BR> </FONT></STRONG></FONT><A href="http://www.garymgordon.com/flash_samples/demo/logo_flash_sample.html" target="_blank"><FONT face=Verdana color=#000040 size=2>http://www.garymgordon.com/flash_samples/demo/logo_flash_sample.html</FONT></A><BR> <FONT face=Verdana color=#000040 size=2>Simply provide us with any text, logo and/or photo(s) that you would<BR>like to have animated and developed as a FLASH (.SWF) ANIMATION.</FONT></DIV> <DIV><FONT face=Arial size=2></FONT><FONT face=Arial size=2></FONT><FONT face=Arial size=2></FONT><FONT face=Arial size=2></FONT><FONT face=Arial size=2></FONT><FONT face=Arial size=2></FONT><FONT face=Arial size=2></FONT><FONT face=Arial size=2></FONT><BR><FONT face=Verdana color=#000040 size=2> <STRONG>2)</STRONG> <STRONG><FONT color=#ff0000>FREE 'Cursor Trailer - Animated Clock (Javascript)'<BR> </FONT></STRONG></FONT><A href="http://www.garymgordon.com/javascript_samples/clock_cursor/index.html" target="_blank"><FONT face=Verdana color=#000040 size=2>http://www.garymgordon.com/javascript_samples/clock_cursor/index.html</FONT></A><BR> <FONT face=Verdana color=#000040 size=2>We will insert and set up this attractive and eye catching feature to<BR>work on any webpage of your website.</FONT></DIV> <FONT face=Arial size=2></FONT><FONT face=Arial size=2></FONT> <FONT face=Arial size=2></FONT><FONT face=Arial size=2></FONT><FONT face=Arial size=2></FONT><FONT face=Arial size=2></FONT> <DIV><FONT face=Arial size=2></FONT><FONT face=Arial size=2></FONT><FONT face=Arial size=2></FONT><FONT face=Arial size=2></FONT><BR><FONT face=Verdana color=#000040 size=2> <STRONG>3)</STRONG> <STRONG><FONT color=#ff0000>FREE 'your choice of any of the following Javascripts<SPAN class=288043613-22102001>'</SPAN></FONT></STRONG></FONT></DIV> <DIV><FONT size=+0><FONT face=Verdana><FONT color=#000040><FONT size=2><SPAN class=288043613-22102001>W</SPAN>e will insert and set<SPAN class=288043613-22102001> them up</SPAN> to work in any webpage on your website'.</FONT></FONT></FONT></FONT></DIV> <DIV><FONT face=Verdana color=#000040 size=2></FONT> </DIV> <DIV><FONT face=Verdana color=#000040 size=2> <STRONG>a)</STRONG> <STRONG><FONT color=#ff0000><SPAN class=288043613-22102001> </SPAN>Page Content Management</FONT></STRONG> that automatically allows content<BR> to be displayed on a page - <SPAN class=288043613-22102001>from and </SPAN>through a specified date, then</FONT></DIV> <DIV><FONT size=+0><FONT face=Verdana><FONT color=#000040><FONT size=2><SPAN class=288043613-22102001> </SPAN>it <SPAN class=288043613-22102001>is </SPAN>automatically <SPAN class=288043613-22102001>r</SPAN>emoved from the page. <BR> </FONT></FONT></FONT></FONT><A href="http://www.garymgordon.com/javascript_samples/content_mgmt/index.html" target="_blank"><FONT face=Verdana color=#000040 size=2>http://www.garymgordon.com/javascript_samples/content_mgmt/index.html</FONT></A></DIV> <DIV><FONT face=Verdana color=#000040 size=2></FONT> </DIV> <FONT face=Verdana color=#000040 size=2> <STRONG>b)</STRONG> <STRONG><FONT color=#ff0000><SPAN class=288043613-22102001> </SPAN>Auto Scroller</FONT></STRONG> for Text and Images. This provides a<BR> scrolling message that can include text and images (with links<BR> to other pages or websites).<BR> </FONT><A href="http://www.garymgordon.com/javascript_samples/scroller_box/index.html" target="_blank"><FONT face=Verdana color=#000040 size=2>http://www.garymgordon.com/javascript_samples/scroller_box/index.html</FONT></A> <DIV><FONT face=Verdana color=#000040 size=2></FONT> </DIV> <FONT face=Verdana color=#000040 size=2> <STRONG>c)</STRONG> <SPAN class=288043613-22102001><STRONG> </STRONG></SPAN> <STRONG><FONT color=#ff0000>Navigational Menu System</FONT></STRONG> that allows you to utilize a drop<BR> down menu that lists any number of pages, websites, etc. It is<BR> a convenient and space saving way to add lots of links within<BR> a small place on a webpage.<BR> </FONT><A href="http://www.garymgordon.com/javascript_samples/dropdownmenu/index.html" target="_blank"><FONT face=Verdana color=#000040 size=2>http://www.garymgordon.com/javascript_samples/dropdownmenu/index.html</FONT></A> <DIV><FONT face=Verdana color=#000040 size=2></FONT> </DIV> <DIV><FONT face=Arial size=2></FONT><FONT face=Arial size=2></FONT><FONT face=Arial size=2></FONT><FONT face=Arial size=2></FONT><FONT face=Arial size=2></FONT><FONT face=Arial size=2></FONT><BR><FONT face=Verdana color=#000040 size=2><STRONG>Now ... you can choose from any of the above website enhancements<BR>absolutely FREE!</STRONG></FONT></DIV> <DIV><FONT face=Verdana color=#000040 size=2></FONT> </DIV> <DIV><FONT face=Verdana color=#000040 size=2>Then, simply contact us and we will set everything up<SPAN class=288043613-22102001> </SPAN>for you<SPAN class=288043613-22102001> </SPAN><SPAN class=288043613-22102001>-</SPAN> NO CHARGE!</FONT></DIV> <DIV><FONT face=Verdana color=#000040 size=2></FONT> </DIV> <DIV><FONT face=Verdana color=#000040 size=2>(This is simply our way of introducing our company to you.)</FONT></DIV> <DIV><FONT face=Verdana color=#000040 size=2></FONT> </DIV> <DIV><FONT face=Verdana color=#000040 size=2><STRONG>E-mail us at:</STRONG> </FONT><A href="mailto:webmaster@garymgordon.com"><FONT face=Verdana color=#000040 size=2>webmaster@garymgordon.com</FONT></A><BR><FONT face=Verdana color=#000040 size=2>or<BR><STRONG>Phone us at:</STRONG> 1-856-482-8320 (New Jersey, USA)</FONT></DIV> <FONT face=Arial size=2></FONT><FONT face=Arial size=2></FONT> <DIV><FONT face=Arial size=2></FONT><FONT face=Arial size=2></FONT><FONT face=Arial size=2></FONT><FONT face=Arial size=2></FONT><FONT face=Arial size=2></FONT><BR><FONT face=Verdana color=#000040 size=2>==========================================<SPAN class=288043613-22102001>===</SPAN>======</FONT></DIV> <DIV><FONT face=Verdana color=#000040 size=2></FONT><FONT face=Verdana color=#000040 size=2></FONT> </DIV> <DIV><FONT face=Verdana color=#000040 size=2> To request your FREE website enhancement,<BR> please include the following two lines of<BR> information in the BODY of your reply email:</FONT></DIV> <DIV><FONT face=Verdana color=#000040 size=2></FONT> </DIV> <DIV><FONT face=Verdana color=#000040 size=2> </FONT><STRONG><FONT face=Verdana color=#000040 size=2>FREE WEBSITE ENHANCEMENT OFFER<BR> <SPAN class=288043613-22102001> </SPAN><SPAN class=288043613-22102001> </SPAN> WE-149-359-1001</FONT></STRONG></DIV> <DIV><FONT face=Verdana color=#000040 size=2></FONT> </DIV> <DIV><FONT face=Verdana color=#000040 size=2> Also provide us with the following:</FONT></DIV> <DIV><FONT face=Verdana color=#000040 size=2></FONT> </DIV> <DIV><FONT face=Verdana color=#000040 size=2> 1) Your company name and address.<BR> 2) Your first and last name.<BR> 3) Your title/position with the company.<BR> 4) Your telephone number - and the best<BR> time to reach you by phone.<BR> 5) Your website URL (if you have one).</FONT></DIV> <DIV><FONT face=Verdana color=#000040 size=2></FONT> </DIV> <DIV><FONT face=Verdana color=#000040 size=2> We will email <SPAN class=288043613-22102001>or call </SPAN>you back shortly to discuss<BR> the free enhancement of your choice. </FONT></DIV> <DIV><FONT face=Verdana color=#000040 size=2></FONT> </DIV> <DIV><FONT face=Verdana color=#000040 size=2>==========================================<SPAN class=288043613-22102001>===</SPAN>======</FONT><FONT face=Verdana color=#000040 size=2></FONT></DIV> <DIV><FONT face=Verdana color=#000040 size=2></FONT> </DIV> <DIV><FONT face=Verdana color=#000040 size=2> <STRONG>OUR CORPORATE INFORMATION:</STRONG></FONT></DIV> <DIV><FONT face=Verdana color=#000040 size=2></FONT> </DIV> <DIV><FONT face=Verdana color=#000040 size=2> Gary M. Gordon, LLC<BR> - A Global Web Development Corporation<BR> 421 Jamaica Drive<BR> Cherry Hill, New Jersey 08002</FONT></DIV> <DIV><FONT face=Verdana color=#000040 size=2></FONT> </DIV> <DIV><FONT face=Verdana color=#000040 size=2> (Available - 7 days a week.)</FONT></DIV> <DIV><FONT face=Verdana color=#000040 size=2></FONT> </DIV> <DIV><FONT face=Verdana color=#000040 size=2> E-mail :: </FONT><A href="mailto:webmaster@garymgordon.com"><FONT face=Verdana color=#000040 size=2>webmaster@garymgordon.com</FONT></A><BR><FONT face=Verdana color=#000040 size=2> Phone :: 856-482-8320<BR> Fax :: 801-858-4271</FONT></DIV> <DIV><FONT face=Verdana color=#000040 size=2></FONT> </DIV> <DIV><FONT face=Arial size=2></FONT><FONT face=Arial size=2></FONT><FONT face=Arial size=2></FONT><FONT face=Arial size=2></FONT><BR><FONT face=Verdana color=#000040 size=2> Background Information of GARY M. GORDON (Owner):</FONT></DIV> <DIV><FONT face=Verdana color=#000040 size=2></FONT> </DIV> <DIV><FONT face=Verdana color=#000040 size=2> Certified Senior Web Developer since 1996<BR> Certified Web Master Instructor since 1999<BR> 20+ years experience in Sales/Marketing/Graphic Design<BR> and Development</FONT></DIV> <DIV><FONT face=Verdana color=#000040 size=2></FONT> </DIV> <DIV><FONT face=Verdana color=#000040 size=2> <BR> Additional Comments:</FONT></DIV> <DIV><FONT face=Verdana color=#000040 size=2></FONT> </DIV> <DIV><FONT face=Verdana color=#000040 size=2> Gary M. Gordon, LLC (Web Development Services) was created<BR> in an effort to provide quality 'web development' work on a<BR> world-wide scale through the joint efforts of a comprehensive<BR> and world-wide network (that includes a selected group of<BR> professional and certified application programmers, website<BR> developers, graphic designers, etc. - from the USA, Canada,<BR> India, Romania, Japan, Australia, and others).</FONT></DIV> <DIV><FONT face=Verdana color=#000040 size=2></FONT> </DIV> <DIV><FONT color=#000040><FONT face=Verdana size=2> <SPAN class=288043613-22102001> </SPAN> </FONT><STRONG><FONT face=Verdana color=#ff0000 size=2>ALL OF OUR WORK IS PROVIDED<BR> WITH A 100% SATISFACTION GUARANTEE!</FONT></STRONG></FONT></DIV> <DIV><FONT face=Verdana color=#000040 size=2></FONT> </DIV> <DIV><FONT face=Verdana color=#000040 size=2> So, if you're not satisfied with our work, ... you don't pay.<BR> It's as simple as that!</FONT></DIV> <DIV><FONT face=Verdana color=#000040 size=2></FONT> </DIV> <DIV><FONT face=Verdana color=#000040 size=2> Please feel free to visit our website or email us at your<BR> convenience, and we will be happy to help you in any way<BR> possible.</FONT></DIV> <DIV><FONT face=Verdana color=#000040 size=2></FONT> </DIV> <DIV><FONT face=Arial size=2></FONT><FONT face=Arial size=2></FONT><FONT face=Arial size=2></FONT><FONT face=Arial size=2></FONT><FONT face=Arial size=2></FONT><FONT face=Arial size=2></FONT><BR><FONT face=Verdana color=#000040 size=2> * Certain restrictions or requirements may apply to all work<BR> requests, free offers, etc. Please ask for complete details.</FONT></DIV> <DIV><FONT face=Verdana color=#000040 size=2></FONT> </DIV> <DIV><FONT face=Verdana color=#000040 size=2> Free offers are also limited to one per website domain and/or<BR> company. Gary M. Gordon, LLC reserves the right to discontinue,<BR> change, or modify this free offer at any time without notice.</FONT></DIV> <DIV><FONT face=Verdana color=#000040 size=2></FONT> </DIV><FONT face=Arial size=2></FONT> <DIV><FONT face=Arial size=2></FONT><FONT face=Arial size=2></FONT><FONT face=Arial size=2></FONT><BR> </div> <hr solid color="#336699" width="475" align="left"> <br> If your browser doesn't support the display of HTML, you can access the same information as shown above by following the link to take advange of our Free Website Enhancement Offer at: http://www.garymgordon.com/services.html (then, on this page, look for the link that says "Free Current Promotional Offer). <br> <br> </BODY></HTML> <br>-------------------------------------------------------------------------------------<br> <font face='arial,verdana' size='1' color='#336699'>If you have joined Gary M. Gordon, LLC by accident or someone else has joined you without your permission,<br> or if you ever want to remove yourself from Gary M. Gordon, LLC, simply visit:<br> <a href='http://www.garymgordon.com/easylist/easylist.cgi?action=unsubscribe&submitemail=python-dev@python.org'>http://www.garymgordon.com/easylist/easylist.cgi?action=unsubscribe&submitemail=python-dev@python.org</a><br> and you will be automatically removed.</font><br> From guido@python.org Fri Oct 26 15:32:59 2001 From: guido@python.org (Guido van Rossum) Date: Fri, 26 Oct 2001 10:32:59 -0400 Subject: [Python-Dev] innocent question ;-) In-Reply-To: Your message of "Fri, 26 Oct 2001 16:23:31 +0200." <20011026142331.C8E84303181@snelboot.oratrix.nl> References: <20011026142331.C8E84303181@snelboot.oratrix.nl> Message-ID: <200110261432.KAA18981@cj20424-a.reston1.va.home.com> > At what point can we consider the API stable? All this inheriting > and subtyping seems like it would be absolutely brilliant for the > Mac toolbox APIs (and I suspect also for the PythonWin API's, Mark?) > but I've refrained from even looking at it (well, I did have a quick > peek:-), because (a) I have no time, (b) I don't want to chase a > changing API and (c) there's some methodchain-based > poor-mans-inheritance in there already that will undoubtedly break. I don't plan to change the C API that was released in 2.2b1 incompatibly. Besides, I won't have time to touch it after next Friday. :-) --Guido van Rossum (home page: http://www.python.org/~guido/) From barry@zope.com Fri Oct 26 15:45:13 2001 From: barry@zope.com (Barry A. Warsaw) Date: Fri, 26 Oct 2001 10:45:13 -0400 Subject: [Python-Dev] iterator support for SRE? References: <014801c15cd6$3d18b270$ced241d5@hagrid> <200110250335.XAA10393@cj20424-a.reston1.va.home.com> <015a01c15e01$f99f4ef0$0900a8c0@spiff> Message-ID: <15321.30457.83985.982168@anthem.wooz.org> >>>>> "FL" == Fredrik Lundh <fredrik@pythonware.com> writes: FL> as you might have noticed, I went ahead and checked in code FL> for "finditer". nobody has complained this far ;-) Please add something to Misc/NEWS file too! -Barry From martin@v.loewis.de Fri Oct 26 19:31:50 2001 From: martin@v.loewis.de (Martin v. Loewis) Date: Fri, 26 Oct 2001 20:31:50 +0200 Subject: [Python-Dev] Re: patchlevel.h In-Reply-To: <20011026102540.3DBEB303181@snelboot.oratrix.nl> (message from Jack Jansen on Fri, 26 Oct 2001 12:25:39 +0200) References: <20011026102540.3DBEB303181@snelboot.oratrix.nl> Message-ID: <200110261831.f9QIVob01347@mira.informatik.hu-berlin.de> > When the release branch is made there's usually a flurry of minor > fixes, and these are almost guaranteed to break something on the > mac. Tiny things, like missing casts, but breaks nonetheless. While this may happen in principle, and out of curiosity: Was this a problem in 2.2b1 also? Looking at the changes made on the 2.2b1 release branch, I see that a total of 7 files was changed. Except for patchlevel.h, and the \n\ fix on socketmodule.c, there were no changes to C code. So I can't see why the changes on the release branch could have any effect on the Mac port, atleast in this release. > Moreover, the "release loop" is now about 24 hours long, according > to PEP101, and even extending it seriously (like to about a week) > still wouldn't guarantee that I could react timely. Not only am I 6 > hours away from the unix/win release folks, but I also have a paying > job that is less and less MacPython-related, so I can't firmly > commit myself. I'm not asking that you work harder on Python, I'm asking that you work less :-) Seriously, I think there is a much larger set of people testing the CVS regularly, so I doubt any breakage atleast on OS X wouldn't be noticed within hours. > And it has happened to me already (twice, I think) that there was a > showstopper bug on the Mac that has caused me to either be very late > with a release or skip one altogether. This happens on the Mac more > often than on Unix/Windows, because MacPython relies on a third > party unix I/O emulation library. I sympathize with this problem, and I can't really suggest a good solution to it. I'm also not so much concerned about concerned about beta releases; nobody will care whether they can build 2.2b1 from the sources on the Mac two months from now. It's just that I think very strict principles must be applied for the final release. If that means that the 2.2 release can't go without ok from you (or somebody else who has produced MacOS 9 binaries), I think we should add that to the release procedure. That check would be to avoid show-stopper bugs only, of course - anything complex needs to be detected long before the final release. Regards, Martin From guido@python.org Fri Oct 26 19:49:28 2001 From: guido@python.org (Guido van Rossum) Date: Fri, 26 Oct 2001 14:49:28 -0400 Subject: [Python-Dev] Re: patchlevel.h In-Reply-To: Your message of "Fri, 26 Oct 2001 20:31:50 +0200." <200110261831.f9QIVob01347@mira.informatik.hu-berlin.de> References: <20011026102540.3DBEB303181@snelboot.oratrix.nl> <200110261831.f9QIVob01347@mira.informatik.hu-berlin.de> Message-ID: <200110261849.OAA20387@cj20424-a.reston1.va.home.com> From guido@python.org Fri Oct 26 19:56:20 2001 From: guido@python.org (Guido van Rossum) Date: Fri, 26 Oct 2001 14:56:20 -0400 Subject: [Python-Dev] Re: patchlevel.h In-Reply-To: Your message of "Fri, 26 Oct 2001 20:31:50 +0200." <200110261831.f9QIVob01347@mira.informatik.hu-berlin.de> References: <20011026102540.3DBEB303181@snelboot.oratrix.nl> <200110261831.f9QIVob01347@mira.informatik.hu-berlin.de> Message-ID: <200110261856.OAA20424@cj20424-a.reston1.va.home.com> > > When the release branch is made there's usually a flurry of minor > > fixes, and these are almost guaranteed to break something on the > > mac. Tiny things, like missing casts, but breaks nonetheless. > > While this may happen in principle, and out of curiosity: Was this a > problem in 2.2b1 also? Looking at the changes made on the 2.2b1 > release branch, I see that a total of 7 files was changed. Except for > patchlevel.h, and the \n\ fix on socketmodule.c, there were no changes > to C code. What's more common is actually a flurry of checkins just *before* the release branch is made (say in the last two days). These often include a somewhat hasty commit of a project that's valuable to have but introduces new bugs -- either Mac-specific or generic. The patch that introduced the missing \n\ problem was an example of this. I don't know what to do about this. Branching several days earlier doesn't prevent it, because usually there's a really good reason to copy the last-minute changes into the release branch. (We don't have the problem that some other projects appear to have, which is that there's a lot of tentative development on the trunk -- then it would make sense to branch earlier, but we tend to use the patch manager for experiments.) > It's just that I think very strict principles must be applied for the > final release. If that means that the 2.2 release can't go without ok > from you (or somebody else who has produced MacOS 9 binaries), I think > we should add that to the release procedure. That check would be to > avoid show-stopper bugs only, of course - anything complex needs to be > detected long before the final release. I think that for the final release we should definitely attempt this. Hopefully the speed and quality of change will reduce closer to the final release. Also, the release candidate should help. Jack, We'll try to work with you. Barry, please add something to this effect in the release procedure PEP (hold up the *final* release until Jack approves or until we lose patience). --Guido van Rossum (home page: http://www.python.org/~guido/) From jim@interet.com Fri Oct 26 20:13:48 2001 From: jim@interet.com (James C. Ahlstrom) Date: Fri, 26 Oct 2001 15:13:48 -0400 Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives Message-ID: <3BD9B5EC.119319A5@interet.com> The PEP for zip import is 273. Please take a look and comment. http://python.sourceforge.net/peps/pep-0273.html Jim Ahlstrom From guido@python.org Fri Oct 26 21:34:15 2001 From: guido@python.org (Guido van Rossum) Date: Fri, 26 Oct 2001 16:34:15 -0400 Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives In-Reply-To: Your message of "Fri, 26 Oct 2001 15:13:48 EDT." <3BD9B5EC.119319A5@interet.com> References: <3BD9B5EC.119319A5@interet.com> Message-ID: <200110262034.QAA20802@cj20424-a.reston1.va.home.com> > The PEP for zip import is 273. Please take a look and comment. > > http://python.sourceforge.net/peps/pep-0273.html OK, I'll shoot. But I expect Gordon McMillan and Greg Stein to provide more useful feedback. | Currently, sys.path is a list of directory names as strings. If | this PEP is implemented, an item of sys.path can be a string | naming a zip file archive. The zip archive can contain a | subdirectory structure to support package imports. The zip | archive satisfies imports exactly as a subdirectory would. I like this. | The implementation is in C code in the Python core and works on | all supported Python platforms. This is good too, as it provides a bootstrap. OTOH I also would like to see a prototype in Python, using either ihooks or imputil. | Any files may be present in the zip archive, but only files *.pyc, | *.pyo and __init__.py[co] are available for import. Zip import of | *.py and dynamic modules (*.pyd, *.so) is disallowed. | | Just as sys.path currently has default directory names, default | zip archive names are added too. Otherwise there is no way to | import all Python library files from an archive. More bootstrap goodness. | Reading compressed zip archives requires the zlib module. An | import of zlib will be attempted prior to any other imports. If | zlib is not available at that time, only uncompressed archives | will be readable, even if zlib subsequently becomes available. Hm, I wonder if we couldn't just link with the libz.a C library and use the C interface, if you're implementing this in C anyway. | Subdirectory Equivalence | | The zip archive must be treated exactly as a subdirectory tree so | we can support package imports based on current and future rules. | Zip archive files must be created with relative path names. That | is, archive file names are of the form: file1, file2, dir1/file3, | dir2/dir3/file4. | | Suppose sys.path contains "/A/B/SubDir" and "/C/D/E/Archive.zip", | and we are trying to import modfoo from the Q package. Then | import.c will generate a list of paths and extensions and will | look for the file. The list of generated paths does not change | for zip imports. (Very clever.) Suppose import.c generates the path | "/A/B/SubDir/Q/R/modfoo.pyc". Then it will also generate the path | "/C/D/E/Archive.zip/Q/R/modfoo.pyc". Finding the SubDir path is | exactly equivalent to finding "Q/R/modfoo.pyc" in the archive. Nice. | Suppose you zip up /A/B/SubDir/* and all its subdirectories. Then | your zip file will satisfy imports just as your subdirectory did. | | Well, not quite. You can't satisfy dynamic modules from a zip | file. Dynamic modules have extensions like .dll, .pyd, and .so. | They are operating system dependent, and probably can't be loaded | except from a file. It might be possible to extract the dynamic | module from the zip file, write it to a plain file and load it. | But that would mean creating temporary files, and dealing with all | the dynload_*.c, and that's probably not a good idea. Agreed. | You also can't import source files *.py from a zip archive. The | problem here is what to do with the compiled files. Python would | normally write these to the same directory as *.py, but surely we | don't want to write to the zip file. We could write to the | directory of the zip archive, but that would clutter it up, not | good if it is /usr/bin for example. We could just fail to write | the compiled files, but that makes zip imports very slow, and the | user would probably not figure out what is wrong. It is probably | best for users to put *.pyc into zip archives in the first place, | and this PEP enforces that rule. I agree. But it would still be good if the .py files were also in the zip file, so the source can be used in tracebacks etc. A C API to get a source line from a filename might be a good idea (plus a Python API). | So the only imports zip archives support are *.pyc and *.pyo, plus | the import of __init__.py[co] for packages, and the search of the | subdirectory structure for the same. I wonder if we need to make an additional rule that allows a .pyc file to satisfy a module request even if we're in optimized mode (where normally only .pyo files are searched). Otherwise, if someone ships a zipfile with only .pyc files, their modules can't be imported at all when python -O is used. | Efficiency | | The only way to find files in a zip archive is linear search. But there's an index record at the end that provides quick access. So | for each zip file in sys.path, we search for its names once, and | put the names plus other relevant data into a static Python | dictionary. The key is the archive name from sys.path joined with | the file name (including any subdirectories) within the archive. | This is exactly the name generated by import.c, and makes lookup | easy. We could do this kind of pre-scanning for regular dictionaries on sys.path too. I found out very long ago (around '93 or '94) that this saves a *lot* of startup time; I presume it still does. (And even more if the info can be cached in a file.) The only problem is how to detect when the cache becomes out of date. Of course, you could say "if you want faster startup time, put all your files in a zip archive", and I couldn't really argue with that. :-) | zlib | | Compressed zip archives require zlib for decompression. Prior to | any other imports, we attempt an import of zlib, and set a flag if | it is available. All compressed files are invisible unless this | flag is true. Do we get an "module not found" error or something better, like "compressed module found as <filename> but zlib unavailable"? | It could happen that zlib was available later. For example, the | import of site.py might add the correct directory to sys.path so a | dynamic load succeeds. But compressed files will still be | invisible. It is unknown if it can happen that importing site.py | can cause zlib to appear, so maybe we're worrying about nothing. | On Windows and Linux, the early import of zlib succeeds without | site.py. Yes, site.py isn't needed to make standard library modules available; it's intended to make non-standare library modules available. :-) | The problem here is the confusion caused by the reverse. Either a | zip file satisfies imports or it doesn't. It is silly to say that | site.py needs to be uncompressed, and that maybe imports will | succeed later. If you don't like this, create uncompressed zip | archives or make sure zlib is available, for example, as a | built-in module. Or we can write special search logic during zip | initialization. I don't think we need anything special here. site.py shouldn't be needed. | Booting | | Python imports site.py itself, and this imports os, nt, ntpath, | stat, and UserDict. It also imports sitecustomize.py which may | import more modules. Zip imports must be available before site.py | is imported. | | Just as there are default directories in sys.path, there must be | one or more default zip archives too. | | The problem is what the name should be. The name should be linked | with the Python version, so the Python executable can correctly | find its corresponding libraries even when there are multiple | Python versions on the same machine. | | This PEP suggests a zip archive name equal to the Python | interpreter path with extension ".zip" (eg, /usr/bin/python.zip) | which is always prepended to sys.path. So a directory with python | and python.zip is complete. This would work fine on Windows, as | it is common to put supporting files in the directory of the | executable. But it may offend Unix fans, who dislike bin | directories being used for libraries. It might be fine to | generate different defaults for Windows and Unix if necessary, but | the code will be in C, and there is no sense getting complicated. Well, this is the domain of getpath.c, and that's got a different implementation for Unix and Windows anyway (Windows has PC/getpathp.c). | Implementation | | A C implementation exists which works, but which can be made better. Upload as a patch please? --Guido van Rossum (home page: http://www.python.org/~guido/) From bckfnn@worldonline.dk Fri Oct 26 23:22:05 2001 From: bckfnn@worldonline.dk (Finn Bock) Date: Fri, 26 Oct 2001 22:22:05 GMT Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives In-Reply-To: <3BD9B5EC.119319A5@interet.com> References: <3BD9B5EC.119319A5@interet.com> Message-ID: <3bd9d419.10386565@mail.wanadoo.dk> [James C. Ahlstrom] >The PEP for zip import is 273. Please take a look and comment. > > http://python.sourceforge.net/peps/pep-0273.html > Any files may be present in the zip archive, but only files *.pyc, > *.pyo and __init__.py[co] are available for import. Zip import of > *.py and dynamic modules (*.pyd, *.so) is disallowed. Why can't .py be allowed? If a more recent .py[co] (or $py.class) exists, it is used. Otherwise the .py file is compiled and discarded when the process ends. Sure, it is slower, but a zip files with only .py[co] entries would be of little use with jython. > Just as sys.path currently has default directory names, default > zip archive names are added too. Otherwise there is no way to > import all Python library files from an archive. A standard for this would be really cool. > Suppose sys.path contains "/A/B/SubDir" and "/C/D/E/Archive.zip", > and we are trying to import modfoo from the Q package. Then > import.c will generate a list of paths and extensions and will > look for the file. The list of generated paths does not change > for zip imports. Suppose import.c generates the path > "/A/B/SubDir/Q/R/modfoo.pyc". Then it will also generate the path > "/C/D/E/Archive.zip/Q/R/modfoo.pyc". Finding the SubDir path is > exactly equivalent to finding "Q/R/modfoo.pyc" in the archive. So python packages and modules can exists *only* at the top level? That would conflict somewhat with jython where at would be common to put python modules into the same .zip file as java classes and java classes also wants to own the root of a zip file. In the current implementaion in jython, we can put python modules under any path and add the zipfile!path to sys.path: sys.path.append("/path/to/zipfile.zip!Lib") which will look for Lib/Q/R/modfoo.py (and Lib/Q/R/modfoo$py.class) in the archive. >Efficiency > The only way to find files in a zip archive is linear search. So > for each zip file in sys.path, we search for its names once, and > put the names plus other relevant data into a static Python > dictionary. The key is the archive name from sys.path joined with > the file name (including any subdirectories) within the archive. > This is exactly the name generated by import.c, and makes lookup > easy. [I found efficiency hard to achieve in jython because opening a zipfile in java also cause the zip index to be read into a dictionary. So we did not want to reopen a zipfile if it can be avoided. Instead we hide a reference to the opened file in sys.path so that when removing a zipfile name from sys.path, the file is eventually closed.] Would entries in the static python dict be removed when a zipfile is removed from sys.path? What is the __path__ vrbl set to in a module imported from a zipfile? Can the module make changes to __path__ and will be changes to used when importing submodules? What value should __file__ have? regards, finn From gerhard@bigfoot.de Sat Oct 27 00:42:12 2001 From: gerhard@bigfoot.de (Gerhard =?iso-8859-1?Q?H=E4ring?=) Date: Sat, 27 Oct 2001 01:42:12 +0200 Subject: [Python-Dev] Future of SSL Message-ID: <20011027014211.A11092@lilith.hqd-internal> First I'd like to say hello. I'm new to this list :-) I've some concerns with the latest development wrt SSL development in Python. The SSL support in the Python core seems to have been added with Python 2.0, and Guido told me in a SF patch discussion, that Pythonlabs doesn't know the code very well. Meanwhile it seems like Jeremy has tried fixing bugs. I've also submitted several patches to the SSL code. As you might know, there are several third-party SSL implementatations for Python (m2crypto, pyOpenSSL, POW). I've contacted the maintainers of these packages and asked them if they'd like to review my patches. The two that answered told me that the reason they implemented their own packages was that the SSL support in Python itself was very incomplete and, to sum it up, Python's SSL is just plain broken. The current state is that there is a single ssl() method in the socket module that can be used to instantiate client-side SSL connections. There's also a patch on Sourceforge that adds server-side SSL functionality. Several points why the current API is broken: - no support for specifying the SSL protocol (SSL v2/SSL v3/SSL v2|3) - no proper certicate checking - SSL objects are neither compatible with sockets nor are they file-like objects - no advanced features like SSL callbacks (to allow the user to accept a cerficicate, for example) What I'd suggest for Python 2.2 is to *not* add any new features, like server-side SSL but only accept bugfixes for the current client-side code. As the current implementation is broken and there is probably little SSL knowledge in the Python core team, I propose to "outsource" the problem: It should be possible to define a "Python SSL interface" that describes an API for SSL. The various modules in Python that use SSL (urllib, smtp, ...) would then be rewritten to use the new API. The socketmodule.c would be rewritten to use the new API instead. Then, wrappers could be written for the various SSL modules that wrap them into the new "Python SSL interface" API. As for me, I'm not an expert in SSL, but I'd be willing to try coordinate the efforts, write a PEP, talk with the module maintainers and such. I'd be grateful to hear your opinions about this newbie proposal :-) Gerhard -- mail: gerhard <at> bigfoot <dot> de registered Linux user #64239 web: http://www.cs.fhm.edu/~ifw00065/ OpenPGP public key id 86AB43C0 public key fingerprint: DEC1 1D02 5743 1159 CD20 A4B6 7B22 6575 86AB 43C0 reduce(lambda x,y:x+y,map(lambda x:chr(ord(x)^42),tuple('zS^BED\nX_FOY\x0b'))) From guido@python.org Sat Oct 27 03:23:30 2001 From: guido@python.org (Guido van Rossum) Date: Fri, 26 Oct 2001 22:23:30 -0400 Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives In-Reply-To: Your message of "Fri, 26 Oct 2001 22:22:05 GMT." <3bd9d419.10386565@mail.wanadoo.dk> References: <3BD9B5EC.119319A5@interet.com> <3bd9d419.10386565@mail.wanadoo.dk> Message-ID: <200110270223.WAA23737@cj20424-a.reston1.va.home.com> > > Just as sys.path currently has default directory names, default > > zip archive names are added too. Otherwise there is no way to > > import all Python library files from an archive. > > A standard for this would be really cool. Yup. > > Suppose import.c generates the path > > "/A/B/SubDir/Q/R/modfoo.pyc". Then it will also generate the path > > "/C/D/E/Archive.zip/Q/R/modfoo.pyc". Finding the SubDir path is > > exactly equivalent to finding "Q/R/modfoo.pyc" in the archive. > > So python packages and modules can exists *only* at the top level? That > would conflict somewhat with jython where at would be common to put > python modules into the same .zip file as java classes and java classes > also wants to own the root of a zip file. > > In the current implementaion in jython, we can put python modules under > any path and add the zipfile!path to sys.path: > > sys.path.append("/path/to/zipfile.zip!Lib") > > which will look for Lib/Q/R/modfoo.py (and Lib/Q/R/modfoo$py.class) in > the archive. Maybe it's possible to allow "/path/to/zipfile.zip/subdirectory/etc/" in sys.path? That sounds better than picking a random new character as delimiter. > [I found efficiency hard to achieve in jython because opening a zipfile > in java also cause the zip index to be read into a dictionary. So we > did not want to reopen a zipfile if it can be avoided. Instead we hide a > reference to the opened file in sys.path so that when removing a zipfile > name from sys.path, the file is eventually closed.] I don't think Python has this problem, since we have control over the zipfile reading code. > Would entries in the static python dict be removed when a zipfile is > removed from sys.path? It can be arranged that they are removed at some later point (e.g. when sys.path is next searched). > What is the __path__ vrbl set to in a module imported from a zipfile? > Can the module make changes to __path__ and will be changes to used when > importing submodules? > > What value should __file__ have? IMO these two questions are answered by the pathname semantics that Jim proposes: __file__ = "/C/D/E/Archive.zip/Q/R/modfoo.pyc" and __path__ = ["/C/D/E/Archive.zip/Q/R"]. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Sat Oct 27 03:26:38 2001 From: guido@python.org (Guido van Rossum) Date: Fri, 26 Oct 2001 22:26:38 -0400 Subject: [Python-Dev] Future of SSL In-Reply-To: Your message of "Sat, 27 Oct 2001 01:42:12 +0200." <20011027014211.A11092@lilith.hqd-internal> References: <20011027014211.A11092@lilith.hqd-internal> Message-ID: <200110270226.WAA23767@cj20424-a.reston1.va.home.com> > What I'd suggest for Python 2.2 is to *not* add any new features, like > server-side SSL but only accept bugfixes for the current client-side > code. Sounds good to me. > As the current implementation is broken and there is probably little SSL > knowledge in the Python core team, I propose to "outsource" the problem: Thanks! We can sure use some help here. > It should be possible to define a "Python SSL interface" that describes > an API for SSL. The various modules in Python that use SSL (urllib, > smtp, ...) would then be rewritten to use the new API. The > socketmodule.c would be rewritten to use the new API instead. I've just started digging in the socketmodule.c for a different reason, and I propose to move all the SSL stuff to a separate file and module. > Then, wrappers could be written for the various SSL modules that wrap > them into the new "Python SSL interface" API. This is a good idea. The DB API works like this. > As for me, I'm not an expert in SSL, but I'd be willing to try > coordinate the efforts, write a PEP, talk with the module maintainers > and such. But we do need *an* expert, don't we? Maybe you can develop expertise as you go? > I'd be grateful to hear your opinions about this newbie proposal :-) You don't sound much like a newbie. :-) --Guido van Rossum (home page: http://www.python.org/~guido/) From petrilli@amber.org Sat Oct 27 04:55:27 2001 From: petrilli@amber.org (Christopher Petrilli) Date: Fri, 26 Oct 2001 23:55:27 -0400 Subject: [Python-Dev] Future of SSL In-Reply-To: <200110270226.WAA23767@cj20424-a.reston1.va.home.com>; from guido@python.org on Fri, Oct 26, 2001 at 10:26:38PM -0400 References: <20011027014211.A11092@lilith.hqd-internal> <200110270226.WAA23767@cj20424-a.reston1.va.home.com> Message-ID: <20011026235527.A2919@trump.amber.org> Guido van Rossum [guido@python.org] wrote: > > Then, wrappers could be written for the various SSL modules that wrap > > them into the new "Python SSL interface" API. > > This is a good idea. The DB API works like this. I think that if OpenSSL is available, Python should build "out of the box" with SSL support. This is becomming more and more important with projects I'm working on, especially with SOAP and XML-RPC. This doesn't mean someone shouldn't be able to replace it, and we should always define an API, but... I think we need to work out of the box. > > As for me, I'm not an expert in SSL, but I'd be willing to try > > coordinate the efforts, write a PEP, talk with the module maintainers > > and such. > > But we do need *an* expert, don't we? Maybe you can develop expertise > as you go? I don't have time to provide code right now, but I do know SSL and X.509 specifically inside and out and would be happy to provide support from a standards/crypto/certificate perspective. Chris -- | Christopher Petrilli | petrilli@amber.org From guido@python.org Sat Oct 27 16:56:34 2001 From: guido@python.org (Guido van Rossum) Date: Sat, 27 Oct 2001 11:56:34 -0400 Subject: [Python-Dev] Future of SSL In-Reply-To: Your message of "Fri, 26 Oct 2001 23:55:27 EDT." <20011026235527.A2919@trump.amber.org> References: <20011027014211.A11092@lilith.hqd-internal> <200110270226.WAA23767@cj20424-a.reston1.va.home.com> <20011026235527.A2919@trump.amber.org> Message-ID: <200110271556.LAA25312@cj20424-a.reston1.va.home.com> > I think that if OpenSSL is available, Python should build "out of the > box" with SSL support. This is becomming more and more important with > projects I'm working on, especially with SOAP and XML-RPC. This > doesn't mean someone shouldn't be able to replace it, and we should > always define an API, but... I think we need to work out of the box. Good point. That's how the SSL support is configured now, and that's how it should continue to work. (Note that, alas, the DB-API never made it this far -- we still haven't been able to find the time to do the tedious work of moving the various database adapters in the core distribution. :-( ) --Guido van Rossum (home page: http://www.python.org/~guido/) From tim.one@home.com Sat Oct 27 19:41:41 2001 From: tim.one@home.com (Tim Peters) Date: Sat, 27 Oct 2001 14:41:41 -0400 Subject: [Python-Dev] dict comps In-Reply-To: <3BD9268A.72AD71E9@lemburg.com> Message-ID: <LNBBLJKPBEHFEDALKOLCMEHDMDAA.tim.one@home.com> [M.-A. Lemburg] > ... > Seriously, this goes down the path of lazy evaluation of expressions. Lazy sequences specifically, but that's been a tension in Python all along (it started with the for/__getitem__ protocol, and nothing we do from here on in will ever be as much of a hack as xrange() was for that <0.9 wink>). > Not sure whether this is the right path to follow though (can cause > brain damage due to overloaded builtin debuggers, IMHO). We can introduce "L[...]" for explicitly lazy list comprehenions <wink>. > BTW, looks like I can finally get rid off the dict() builtin I > have in mxTools which is Good News ! It's not quite the same in the details: CVS dictionary(s) works just like CVS d = {} for k, v in s: d[k] = v In particular, it demands that the elements of s each produce exactly 2 objects, where IIRC the mxTools dict() requires at least 2 (ignoring any after the second). Was ignoring excess objects "a feature", or just exposing an implementation that didn't want to bother to check? From tim.one@home.com Sat Oct 27 19:55:36 2001 From: tim.one@home.com (Tim Peters) Date: Sat, 27 Oct 2001 14:55:36 -0400 Subject: [Python-Dev] iterator support for SRE? In-Reply-To: <015a01c15e01$f99f4ef0$0900a8c0@spiff> Message-ID: <LNBBLJKPBEHFEDALKOLCKEHEMDAA.tim.one@home.com> +1 on finditer. [/F] > ... > will have to think about scanner; I agree that it may seem to > be vaguely useful, but cannot think of a use case that isn't > better handled by findall/split/sub or iterfind. can anyone? It's in desperate need of usage docs. The few times I recall seeing it mentioned on c.l.py, it was from someone staring at the code unable to reverse-engineer its intended use. Therefore I don't believe it's gotten a good trial, and so I'm at best -0 on making it official now. In my own stuff I've built directly on top of .lastindex/.lastgroup instead -- perhaps only because it's easier to remember how to use your own undocumented stuff than somebody else's <wink>. the-folks-most-likely-to-use-a-framework-are-those-most-likely-to- write-their-own-ly y'rs - tim From tim.one@home.com Sat Oct 27 20:45:03 2001 From: tim.one@home.com (Tim Peters) Date: Sat, 27 Oct 2001 15:45:03 -0400 Subject: [Python-Dev] Idle speed freak looking for trouble? Message-ID: <LNBBLJKPBEHFEDALKOLCAEHHMDAA.tim.one@home.com> SF bug 474992 contains a simple benchmark claimed to show dramatic slowdown from 1.5.2 to 2.0, and again but less so from 2.0 to 2.1. Also major slowdown from enabling gc. I *believe* Frederic ran it on Tru64. no-time-here-ly y'rs - tim From fredrik@pythonware.com Sat Oct 27 21:45:36 2001 From: fredrik@pythonware.com (Fredrik Lundh) Date: Sat, 27 Oct 2001 22:45:36 +0200 Subject: [Python-Dev] Idle speed freak looking for trouble? References: <LNBBLJKPBEHFEDALKOLCAEHHMDAA.tim.one@home.com> Message-ID: <004501c15f28$56148340$ced241d5@hagrid> tim wrote: > SF bug 474992 contains a simple benchmark claimed to show dramatic slowdown > from 1.5.2 to 2.0, and again but less so from 2.0 to 2.1. fwiw, here's the timings I get on a slow windows box (using PythonWare builds, and time.clock() instead of os.times()): python 1.5: 2.01 python 2.0: 3.02 python 2.1: 3.00 python 2.2: 3.04 after playing some more, I ended up with this little benchmark: def dosomething(item): return [None, None, None] data = [None] * 100000 import time t1 = time.clock(); x = map(dosomething, data) t2 = time.clock() print t2 - t1 python 1.5: 1.02 seconds python 2.0: 1.95 seconds python 2.1: 1.85 seconds python 2.2: 1.90 seconds </F> From mal@lemburg.com Sat Oct 27 21:38:03 2001 From: mal@lemburg.com (M.-A. Lemburg) Date: Sat, 27 Oct 2001 22:38:03 +0200 Subject: [Python-Dev] Idle speed freak looking for trouble? References: <LNBBLJKPBEHFEDALKOLCAEHHMDAA.tim.one@home.com> Message-ID: <3BDB1B2B.8705CCCE@lemburg.com> Tim Peters wrote: > > SF bug 474992 contains a simple benchmark claimed to show dramatic slowdown > from 1.5.2 to 2.0, and again but less so from 2.0 to 2.1. Also major > slowdown from enabling gc. I *believe* Frederic ran it on Tru64. ... want me to run pybench against the different versions ? -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Consulting & Company: http://www.egenix.com/ Python Software: http://www.lemburg.com/python/ From mal@lemburg.com Sat Oct 27 21:53:04 2001 From: mal@lemburg.com (M.-A. Lemburg) Date: Sat, 27 Oct 2001 22:53:04 +0200 Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives References: <3BD9B5EC.119319A5@interet.com> Message-ID: <3BDB1EB0.C2A9B1FF@lemburg.com> "James C. Ahlstrom" wrote: > > The PEP for zip import is 273. Please take a look and comment. > > http://python.sourceforge.net/peps/pep-0273.html Looks good to me. Just three questions: 1. Why are .py files being outlawed ? 2. Where's the C implementation you mention in the PEP ? 3. Would it be possible to ship zlib together with Python ? (the zlib license should allow this and I don't think that the code size is too big) -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Consulting & Company: http://www.egenix.com/ Python Software: http://www.lemburg.com/python/ From mal@lemburg.com Sat Oct 27 21:46:04 2001 From: mal@lemburg.com (M.-A. Lemburg) Date: Sat, 27 Oct 2001 22:46:04 +0200 Subject: [Python-Dev] dict comps References: <LNBBLJKPBEHFEDALKOLCMEHDMDAA.tim.one@home.com> Message-ID: <3BDB1D0C.A60F2EF@lemburg.com> Tim Peters wrote: > > [M.-A. Lemburg] > > ... > > Seriously, this goes down the path of lazy evaluation of expressions. > > Lazy sequences specifically, but that's been a tension in Python all along > (it started with the for/__getitem__ protocol, and nothing we do from here > on in will ever be as much of a hack as xrange() was for that <0.9 wink>). > > > Not sure whether this is the right path to follow though (can cause > > brain damage due to overloaded builtin debuggers, IMHO). > > We can introduce "L[...]" for explicitly lazy list comprehenions <wink>. ... and add a lazy list comprehension object (e.g. xlistcompobj) ? Cool :-) > > BTW, looks like I can finally get rid off the dict() builtin I > > have in mxTools which is Good News ! > > It's not quite the same in the details: CVS dictionary(s) works just like > CVS > > d = {} > for k, v in s: > d[k] = v > > In particular, it demands that the elements of s each produce exactly 2 > objects, where IIRC the mxTools dict() requires at least 2 (ignoring any > after the second). Was ignoring excess objects "a feature", or just > exposing an implementation that didn't want to bother to check? It was a feature; it's just that I forgot what I needed it for ;-) There are quite a few things in mxTools which I've never *really* needed. The single most used API in the package was and still is irange() which returns a sequence (i, obj[i]). dict() and invdict() are also rather popular ones. Most of the others tend to solve performance problems in some inner loops of some applications I wrote. -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Consulting & Company: http://www.egenix.com/ Python Software: http://www.lemburg.com/python/ From fredrik@pythonware.com Sat Oct 27 22:21:56 2001 From: fredrik@pythonware.com (Fredrik Lundh) Date: Sat, 27 Oct 2001 23:21:56 +0200 Subject: [Python-Dev] Idle speed freak looking for trouble? Message-ID: <006201c15f2d$6b986100$ced241d5@hagrid> I wrote: > python 1.5: 1.02 seconds > python 2.0: 1.95 seconds > python 2.1: 1.85 seconds > python 2.2: 1.90 seconds doh. forgot to disable GC for 2.0 and later. with GC switched off, the difference is a bit smaller. python 1.5: 1.02 seconds python 2.0: 1.30 seconds python 2.1: 1.20 seconds python 2.2: 1.05 seconds now replace [None, None, None] with [item, item, item]: python 1.5: 0.58 seconds python 2.0: 0.85 seconds python 2.1: 0.69 seconds python 2.2: 0.71 seconds playing with other dosomething bodies (and with GC switched off), I've noticed that the following things has gotten slower from 1.5.2 to 2.2: - concatenating strings (item+item+item is 20% slower) - calling the float builtin using global lookup (20% slower) - list and tuple forming using locals (10-20% slower) </F> From petrilli@amber.org Sun Oct 28 01:43:58 2001 From: petrilli@amber.org (Christopher Petrilli) Date: Sat, 27 Oct 2001 21:43:58 -0400 Subject: [Python-Dev] Future of SSL In-Reply-To: <200110271556.LAA25312@cj20424-a.reston1.va.home.com>; from guido@python.org on Sat, Oct 27, 2001 at 11:56:34AM -0400 References: <20011027014211.A11092@lilith.hqd-internal> <200110270226.WAA23767@cj20424-a.reston1.va.home.com> <20011026235527.A2919@trump.amber.org> <200110271556.LAA25312@cj20424-a.reston1.va.home.com> Message-ID: <20011027214357.B6993@trump.amber.org> Guido van Rossum [guido@python.org] wrote: > > I think that if OpenSSL is available, Python should build "out of the > > box" with SSL support. This is becomming more and more important with > > projects I'm working on, especially with SOAP and XML-RPC. This > > doesn't mean someone shouldn't be able to replace it, and we should > > always define an API, but... I think we need to work out of the box. > > Good point. That's how the SSL support is configured now, and that's > how it should continue to work. Perhaps there is one of the existing modules (M2Crypto?) that can be integrated, assuming licensing issues can be resolved. Also, I think that perhaps high level abstractions should be introduced, though I'm not sure what they are now... that's just hand waving. The initial goal in my mind would be to have transparent (or nearly so) SSL session management, but the X.509 manipulation and general crypto interface could wait until later. While they are both useful, the SSL side is the really critical part. > (Note that, alas, the DB-API never made it this far -- we still > haven't been able to find the time to do the tedious work of moving > the various database adapters in the core distribution. :-( ) Actually what this looks more like is not just SSL, but a "crypto" collection for Python, dependent on the excellent work in OpenSSL. I can start outlining some stuff if that would be a good start, though obviously if there's an existing library that could be integrated, that would be excellent. Chris -- | Christopher Petrilli | petrilli@amber.org From michel@zope.com Sun Oct 28 02:02:42 2001 From: michel@zope.com (Michel Pelletier) Date: Sat, 27 Oct 2001 19:02:42 -0700 Subject: [Python-Dev] Future of SSL References: <20011027014211.A11092@lilith.hqd-internal> <200110270226.WAA23767@cj20424-a.reston1.va.home.com> <20011026235527.A2919@trump.amber.org> <200110271556.LAA25312@cj20424-a.reston1.va.home.com> <20011027214357.B6993@trump.amber.org> Message-ID: <3BDB6742.E143BF8B@zope.com> Christopher Petrilli wrote: > > Actually what this looks more like is not just SSL, but a "crypto" > collection for Python, dependent on the excellent work in OpenSSL. I > can start outlining some stuff if that would be a good start, though > obviously if there's an existing library that could be integrated, > that would be excellent. AMK has such a package: http://www.amk.ca/python/code/crypto.html which served as an exellent introduction to the world of crypto for me. -Michel From guido@python.org Sun Oct 28 12:20:30 2001 From: guido@python.org (Guido van Rossum) Date: Sun, 28 Oct 2001 07:20:30 -0500 Subject: [Python-Dev] Future of SSL In-Reply-To: Your message of "Sat, 27 Oct 2001 21:43:58 EDT." <20011027214357.B6993@trump.amber.org> References: <20011027014211.A11092@lilith.hqd-internal> <200110270226.WAA23767@cj20424-a.reston1.va.home.com> <20011026235527.A2919@trump.amber.org> <200110271556.LAA25312@cj20424-a.reston1.va.home.com> <20011027214357.B6993@trump.amber.org> Message-ID: <200110281220.HAA09692@cj20424-a.reston1.va.home.com> > The initial goal in my mind would be to have transparent (or nearly > so) SSL session management, but the X.509 manipulation and general > crypto interface could wait until later. While they are both useful, > the SSL side is the really critical part. Which is of course exactly the goal of the existing socket.ssl support, primitive though it is. This was paid for in part by HP; their goal was to let urllib support the HTTPS protocol, nothing more, nothing less. And it does that well enough. > > (Note that, alas, the DB-API never made it this far -- we still > > haven't been able to find the time to do the tedious work of moving > > the various database adapters in the core distribution. :-( ) > > Actually what this looks more like is not just SSL, but a "crypto" > collection for Python, dependent on the excellent work in OpenSSL. I > can start outlining some stuff if that would be a good start, though > obviously if there's an existing library that could be integrated, > that would be excellent. I think it would be a big mistake to start from scratch. Remember the GUI SIG? PS. One issue with adding more crypto to Python could be US export issues. It's possible that new export limitations for crypto software are made law by a congress that doesn't understand the issues, and then the US Python distribution could be in trouble (even though our site in the the Netherlands, we build the distributions here in the US). Back at CNRI, we couldn't release the SSL wrappers, which don't contain any crypto code but enable linking with it, before an extensive and expensive legal review, and then we had to wait until after a certain date, at which some of the crypto export restrictions were lifted. --Guido van Rossum (home page: http://www.python.org/~guido/) From bckfnn@worldonline.dk Sun Oct 28 14:30:30 2001 From: bckfnn@worldonline.dk (Finn Bock) Date: Sun, 28 Oct 2001 14:30:30 GMT Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives In-Reply-To: <200110270223.WAA23737@cj20424-a.reston1.va.home.com> References: <3BD9B5EC.119319A5@interet.com> <3bd9d419.10386565@mail.wanadoo.dk> <200110270223.WAA23737@cj20424-a.reston1.va.home.com> Message-ID: <3bdbfda4.3697676@mail.wanadoo.dk> [me] > In the current implementaion in jython, we can put python modules under > any path and add the zipfile!path to sys.path: > > sys.path.append("/path/to/zipfile.zip!Lib") > > which will look for Lib/Q/R/modfoo.py (and Lib/Q/R/modfoo$py.class) in > the archive. [GvR] >Maybe it's possible to allow "/path/to/zipfile.zip/subdirectory/etc/" >in sys.path? That sounds better than picking a random new character >as delimiter. Fine by me. From a java POV the bang ("!") was not random: >>> import java >>> java.lang.Class.getResource(type(1), "/java/lang/Class.class") jar:file:/D:/java/jdk1.4/jre/lib/rt.jar!/java/lang/Class.class >>> The URL library in java can of course open the string above as an input stream. >> Would entries in the static python dict be removed when a zipfile is >> removed from sys.path? > >It can be arranged that they are removed at some later point >(e.g. when sys.path is next searched). An API to do this could be usefull for jython. Right now we depend on the GC thread to close the file. Since files are a limited resource it would be a good thing to have an explicit way to clean up the cached resources. I don't expect the lack of a cleanup method to be a huge problem in real life. >> What is the __path__ vrbl set to in a module imported from a zipfile? >> Can the module make changes to __path__ and will be changes to used when >> importing submodules? >> >> What value should __file__ have? > >IMO these two questions are answered by the pathname semantics >that Jim proposes: __file__ = "/C/D/E/Archive.zip/Q/R/modfoo.pyc" and >__path__ = ["/C/D/E/Archive.zip/Q/R"]. Ok. I assume that as long as some module exists with a __path__ like this, it not possible to clear the cached entries for /C/D/E/Archive.zip. If importing from zip is regarded mainly as a deployment/bootstrapping feature, then the cleanup issues does not exist. I have no problem looking at it as a deployment feature (that is all I need it for myself), I just didn't dare to put such a limit on my jython implementation. regards, finn From andymac@bullseye.apana.org.au Sun Oct 28 07:41:46 2001 From: andymac@bullseye.apana.org.au (Andrew MacIntyre) Date: Sun, 28 Oct 2001 18:41:46 +1100 (EDT) Subject: [Python-Dev] status of OS/2 ports of Python In-Reply-To: <200110252130.f9PLUM329494@odiug.zope.com> Message-ID: <Pine.OS2.4.32.0110281805420.605-100000@central> On Thu, 25 Oct 2001, Guido van Rossum wrote: > > At this stage, my plan is to finalise a binary release of 2.2 before > > submitting a new set of patches, taking into account Martin von Loewis' > > comments and suggestions (much appreciated thank you!) among others. I > > also plan to try and coordinate my patches with those of Michael Muller > > (473749, 474169 & 474500 so far) for the VAC++ build of Python for OS/2. > > > > I will try and review Michael's patches with a view to getting them into > > 2.2, as the VAC++ build has been part of the sourceball for a few > > versions. > > Andy, would it help if you had SourceForge commit privileges? You'd > be obliged to only check in code that doesn't break the build on other > platforms, so you would be required to test anything you want to check > in on another platform (Linux or Windows) before you commit to save > you the embarrassment of breaking the build, but if (as I expect) you > would mostly be adding stuff inside "#ifdef OS2", that shouldn't be a > big burden. It would save you the trouble of uploading patches to > SourceForge and it would save *us* the trouble of reviewing your > patches and checking them in. I've never used CVS, but am prepared to rectify that. I am also prepared to work within the commit rules you specify. I can work with the requirement for testing on another platform if FreeBSD is an acceptable test platform. I would also propose to look after patches to the VAC++ build (such as Michael Muller's), on the basis that although I can't test the patches directly, I can test that they don't affect the rest of the system, and I can work with the submitter to resolve issues that affect OS/2 specific functionality. As to whether having commit privs would help - on the time availability front, no; otherwise more likely than not. > All your code would be owned by the PSF, but that's what you want > anyway, right? Yes. -- Andrew I MacIntyre "These thoughts are mine alone..." E-mail: andymac@bullseye.apana.org.au | Snail: PO Box 370 andymac@pcug.org.au | Belconnen ACT 2616 Web: http://www.andymac.org/ | Australia From gerhard@bigfoot.de Sun Oct 28 17:40:25 2001 From: gerhard@bigfoot.de (Gerhard =?iso-8859-1?Q?H=E4ring?=) Date: Sun, 28 Oct 2001 18:40:25 +0100 Subject: [Python-Dev] Future of SSL In-Reply-To: <3BDB6742.E143BF8B@zope.com>; from michel@zope.com on Sat, Oct 27, 2001 at 07:02:42PM -0700 References: <20011027014211.A11092@lilith.hqd-internal> <200110270226.WAA23767@cj20424-a.reston1.va.home.com> <20011026235527.A2919@trump.amber.org> <200110271556.LAA25312@cj20424-a.reston1.va.home.com> <20011027214357.B6993@trump.amber.org> <3BDB6742.E143BF8B@zope.com> Message-ID: <20011028184024.A1066@lilith.hqd-internal> On Sat, Oct 27, 2001 at 07:02:42PM -0700, Michel Pelletier wrote: > Christopher Petrilli wrote: > > > > Actually what this looks more like is not just SSL, but a "crypto" > > collection for Python, dependent on the excellent work in OpenSSL. I > > can start outlining some stuff if that would be a good start, though > > obviously if there's an existing library that could be integrated, > > that would be excellent. > > AMK has such a package: > > http://www.amk.ca/python/code/crypto.html It doesn't offer SSL, however. I've compiled a list of SSL libraries for Python: http://www.cs.fhm.edu/~ifw00065/pyssl/ The "Python OpenSSL wrappers" are BSD licensed and could probably serve as a good basis for a SSL library in Python itself. It's an early release, that I can crash easily if I want to ;-) Also, it would have to be modified to fit Python's coding standards (basically, the docstrings are all written in DocBook; these would have to be translated to normal text). But it's a working solution to start from. The other BSD licensed one is M2Crypto, which is around much longer, but requires SWIG to build. I think SSL support is not reason enough to require SWIG for Python builds. So this one is possibly out. The third one is LGPL licensed, so it's out immediately. Gerhard -- mail: gerhard <at> bigfoot <dot> de registered Linux user #64239 web: http://www.cs.fhm.edu/~ifw00065/ OpenPGP public key id 86AB43C0 public key fingerprint: DEC1 1D02 5743 1159 CD20 A4B6 7B22 6575 86AB 43C0 reduce(lambda x,y:x+y,map(lambda x:chr(ord(x)^42),tuple('zS^BED\nX_FOY\x0b'))) From gerhard@bigfoot.de Sun Oct 28 17:52:51 2001 From: gerhard@bigfoot.de (Gerhard =?iso-8859-1?Q?H=E4ring?=) Date: Sun, 28 Oct 2001 18:52:51 +0100 Subject: [Python-Dev] Future of SSL In-Reply-To: <20011027214357.B6993@trump.amber.org>; from petrilli@amber.org on Sat, Oct 27, 2001 at 09:43:58PM -0400 References: <20011027014211.A11092@lilith.hqd-internal> <200110270226.WAA23767@cj20424-a.reston1.va.home.com> <20011026235527.A2919@trump.amber.org> <200110271556.LAA25312@cj20424-a.reston1.va.home.com> <20011027214357.B6993@trump.amber.org> Message-ID: <20011028185250.B1066@lilith.hqd-internal> On Sat, Oct 27, 2001 at 09:43:58PM -0400, Christopher Petrilli wrote: > Guido van Rossum [guido@python.org] wrote: > > > I think that if OpenSSL is available, Python should build "out of the > > > box" with SSL support. This is becomming more and more important with > > > projects I'm working on, especially with SOAP and XML-RPC. This > > > doesn't mean someone shouldn't be able to replace it, and we should > > > always define an API, but... I think we need to work out of the box. > > > > Good point. That's how the SSL support is configured now, and that's > > how it should continue to work. Ok. I understand completely "outsourcing" SSL is not an option. So we either build a completely new SSL module or try to integrate an existing one. > Perhaps there is one of the existing modules (M2Crypto?) that can be > integrated, assuming licensing issues can be resolved. Yup. To save you time finding them all, I've summarized them and put up a page about them (cf. my other post). > [...] The initial goal in my mind would be to have transparent (or nearly > so) SSL session management, [...] I'm not sure I understand what you mean by transparent session management. Perhaps that one important feature would be that SSL objects be interface compatible with socket objects as much as possible? So ugly hacks like FakeSocket in httplib and SSLFakeSocket in smtplib are no longer necessary. And, btw. one complaint about socketmodule.c I've heard is that it doesn't have a C API, it might be necessary to expose some of it with the help of a header file. Gerhard -- mail: gerhard <at> bigfoot <dot> de registered Linux user #64239 web: http://www.cs.fhm.edu/~ifw00065/ OpenPGP public key id 86AB43C0 public key fingerprint: DEC1 1D02 5743 1159 CD20 A4B6 7B22 6575 86AB 43C0 reduce(lambda x,y:x+y,map(lambda x:chr(ord(x)^42),tuple('zS^BED\nX_FOY\x0b'))) From tim.one@home.com Mon Oct 29 03:14:44 2001 From: tim.one@home.com (Tim Peters) Date: Sun, 28 Oct 2001 22:14:44 -0500 Subject: [Python-Dev] status of OS/2 ports of Python In-Reply-To: <Pine.OS2.4.32.0110281805420.605-100000@central> Message-ID: <LNBBLJKPBEHFEDALKOLCIEKBMDAA.tim.one@home.com> [Andrew MacIntyre, to Guido] > I've never used CVS, but am prepared to rectify that. I am also prepared > to work within the commit rules you specify. > > I can work with the requirement for testing on another platform if FreeBSD > is an acceptable test platform. Yes, FreeBSD is fine. It's "Unix-like", and random collateral damage there is mostly what we're trying to guard against, as most people building from CVS are building on Unix-like systems. > I would also propose to look after patches to the VAC++ build (such as > Michael Muller's), on the basis that although I can't test the patches > directly, I can test that they don't affect the rest of the system, and I > can work with the submitter to resolve issues that affect OS/2 specific > functionality. That would be wonderful. I don't know what to do with OS/2 patches, although since I'm "the Windows guy", people look at me like I'm *supposed* to know something about OS/2 <wink>. I don't. If you and Michael can coordinate, it would help Python on OS/2 a lot. > As to whether having commit privs would help - on the time availability > front, no; otherwise more likely than not. It would be more time-efficient for you (and for us). Waiting for somebody else to review your patches can drag on for-- literally --months, at which point the patch likely doesn't even apply anymore. Checking in directly from a build tree saves a whole lot of time. take-it-easy-on-#ifdefs-outside-of-pyport.h-ly y'rs - tim From guido@python.org Mon Oct 29 07:30:13 2001 From: guido@python.org (Guido van Rossum) Date: Mon, 29 Oct 2001 02:30:13 -0500 Subject: [Python-Dev] status of OS/2 ports of Python In-Reply-To: Your message of "Sun, 28 Oct 2001 22:14:44 EST." <LNBBLJKPBEHFEDALKOLCIEKBMDAA.tim.one@home.com> References: <LNBBLJKPBEHFEDALKOLCIEKBMDAA.tim.one@home.com> Message-ID: <200110290730.CAA11349@cj20424-a.reston1.va.home.com> > > As to whether having commit privs would help - on the time availability > > front, no; otherwise more likely than not. > > It would be more time-efficient for you (and for us). Waiting for somebody > else to review your patches can drag on for-- literally --months, at which > point the patch likely doesn't even apply anymore. Checking in directly > from a build tree saves a whole lot of time. Plus, if you screw up, you have instant review. :-) --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Mon Oct 29 08:15:37 2001 From: guido@python.org (Guido van Rossum) Date: Mon, 29 Oct 2001 03:15:37 -0500 Subject: [Python-Dev] Future of SSL In-Reply-To: Your message of "Sun, 28 Oct 2001 18:52:51 +0100." <20011028185250.B1066@lilith.hqd-internal> References: <20011027014211.A11092@lilith.hqd-internal> <200110270226.WAA23767@cj20424-a.reston1.va.home.com> <20011026235527.A2919@trump.amber.org> <200110271556.LAA25312@cj20424-a.reston1.va.home.com> <20011027214357.B6993@trump.amber.org> <20011028185250.B1066@lilith.hqd-internal> Message-ID: <200110290815.DAA11519@cj20424-a.reston1.va.home.com> > And, btw. one complaint about socketmodule.c I've heard is that it > doesn't have a C API, it might be necessary to expose some of it with > the help of a header file. What would you need from a C API? Socket objects currently are pretty thin wrappers around file descriptors. I'm not against this (although defining and using an external C API for a Python extension module is a bit cumbersome due to the requirement for dynamic loading), but I'd like to see the requirements first. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Mon Oct 29 08:19:41 2001 From: guido@python.org (Guido van Rossum) Date: Mon, 29 Oct 2001 03:19:41 -0500 Subject: [Python-Dev] Future of SSL In-Reply-To: Your message of "Sun, 28 Oct 2001 18:40:25 +0100." <20011028184024.A1066@lilith.hqd-internal> References: <20011027014211.A11092@lilith.hqd-internal> <200110270226.WAA23767@cj20424-a.reston1.va.home.com> <20011026235527.A2919@trump.amber.org> <200110271556.LAA25312@cj20424-a.reston1.va.home.com> <20011027214357.B6993@trump.amber.org> <3BDB6742.E143BF8B@zope.com> <20011028184024.A1066@lilith.hqd-internal> Message-ID: <200110290819.DAA11553@cj20424-a.reston1.va.home.com> > > AMK has such a package: > > > > http://www.amk.ca/python/code/crypto.html > > It doesn't offer SSL, however. I've compiled a list of SSL libraries for > Python: > > http://www.cs.fhm.edu/~ifw00065/pyssl/ > > The "Python OpenSSL wrappers" are BSD licensed and could probably serve > as a good basis for a SSL library in Python itself. It's an early > release, that I can crash easily if I want to ;-) Also, it would have to > be modified to fit Python's coding standards (basically, the docstrings > are all written in DocBook; these would have to be translated to normal > text). But it's a working solution to start from. Do you know the author? > The other BSD licensed one is M2Crypto, which is around much longer, but > requires SWIG to build. I think SSL support is not reason enough to > require SWIG for Python builds. So this one is possibly out. It's definitely out to require SWIG to build Python from a distribution, but we may require SWIG to hack on this particular module; we could ship the SWIG output. Not ideal, but if otherwise this module is the best candidate, maybe we could live with this. > The third > one is LGPL licensed, so it's out immediately. Really? I believe the LGPL is not "viral" as the GPL. Also, there may be the possibility to shame the author into changing the license. Can you review it a bit more technically? --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Mon Oct 29 08:31:16 2001 From: guido@python.org (Guido van Rossum) Date: Mon, 29 Oct 2001 03:31:16 -0500 Subject: [Python-Dev] status of OS/2 ports of Python In-Reply-To: Your message of "Sun, 28 Oct 2001 18:41:46 +1100." <Pine.OS2.4.32.0110281805420.605-100000@central> References: <Pine.OS2.4.32.0110281805420.605-100000@central> Message-ID: <200110290831.DAA11598@cj20424-a.reston1.va.home.com> > > Andy, would it help if you had SourceForge commit privileges? You'd > > be obliged to only check in code that doesn't break the build on other > > platforms, so you would be required to test anything you want to check > > in on another platform (Linux or Windows) before you commit to save > > you the embarrassment of breaking the build, but if (as I expect) you > > would mostly be adding stuff inside "#ifdef OS2", that shouldn't be a > > big burden. It would save you the trouble of uploading patches to > > SourceForge and it would save *us* the trouble of reviewing your > > patches and checking them in. > > I've never used CVS, but am prepared to rectify that. Be prepared for some long evenings of fun with CVS! Lots of folks here on python-dev have developed a deep understanding of CVS theory and tricks though, so you're in a good place to learn. > I am also prepared > to work within the commit rules you specify. I didn't doubt this for a second. > I can work with the requirement for testing on another platform if FreeBSD > is an acceptable test platform. Sure is -- in fact, we could use more FreeBSD testing. :-) > I would also propose to look after patches to the VAC++ build (such as > Michael Muller's), on the basis that although I can't test the patches > directly, I can test that they don't affect the rest of the system, and I > can work with the submitter to resolve issues that affect OS/2 specific > functionality. That sounds like a very good idea. Right now I don't really know what to do with the two sets of OS/2 patches... Since Michael is cc'ed here, Michael, what do you think of this? > As to whether having commit privs would help - on the time availability > front, no; otherwise more likely than not. OK. As soon as you think you're ready to commit anything (try something small first :-), post a message here and one of the admins will add you. (You need to have a SF login first.) Not me, I'm about to go underground for a while to take care of my wife and son (to be born by Friday). > > All your code would be owned by the PSF, but that's what you want > > anyway, right? > > Yes. Good! The PSF legal team will eventually come up with paperwork to make this official; in the mean time, this archived email message is good enough for me. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Mon Oct 29 08:37:00 2001 From: guido@python.org (Guido van Rossum) Date: Mon, 29 Oct 2001 03:37:00 -0500 Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives In-Reply-To: Your message of "Sun, 28 Oct 2001 14:30:30 GMT." <3bdbfda4.3697676@mail.wanadoo.dk> References: <3BD9B5EC.119319A5@interet.com> <3bd9d419.10386565@mail.wanadoo.dk> <200110270223.WAA23737@cj20424-a.reston1.va.home.com> <3bdbfda4.3697676@mail.wanadoo.dk> Message-ID: <200110290837.DAA11621@cj20424-a.reston1.va.home.com> > [GvR] > > >Maybe it's possible to allow "/path/to/zipfile.zip/subdirectory/etc/" > >in sys.path? That sounds better than picking a random new character > >as delimiter. [Finn] > Fine by me. From a java POV the bang ("!") was not random: > > >>> import java > >>> java.lang.Class.getResource(type(1), "/java/lang/Class.class") > jar:file:/D:/java/jdk1.4/jre/lib/rt.jar!/java/lang/Class.class > >>> > > The URL library in java can of course open the string above as an input > stream. I guess the advantage of this notation is that it makes a simple check for "file inside zip archive" possible; I sort of like that. The downside is that it limits the use of "!" for other reasons in pathnames (which seems a mostly but not entirely theoretical problem). > I assume that as long as some module exists with a __path__ like this, > it not possible to clear the cached entries for /C/D/E/Archive.zip. Good point. > If importing from zip is regarded mainly as a deployment/bootstrapping > feature, then the cleanup issues does not exist. I have no problem > looking at it as a deployment feature (that is all I need it for > myself), I just didn't dare to put such a limit on my jython > implementation. I don't see sys.path as a very dynamic thing anyway. It gets manipulated briefly at the beginning, and then maybe, rarely, stuff gets added during the program run. I've seen temporary additions, but these were usually in the test suite. --Guido van Rossum (home page: http://www.python.org/~guido/) From mal@lemburg.com Mon Oct 29 08:36:41 2001 From: mal@lemburg.com (M.-A. Lemburg) Date: Mon, 29 Oct 2001 09:36:41 +0100 Subject: [Python-Dev] Future of SSL References: <20011027014211.A11092@lilith.hqd-internal> <200110270226.WAA23767@cj20424-a.reston1.va.home.com> <20011026235527.A2919@trump.amber.org> <200110271556.LAA25312@cj20424-a.reston1.va.home.com> <20011027214357.B6993@trump.amber.org> <3BDB6742.E143BF8B@zope.com> <20011028184024.A1066@lilith.hqd-internal> Message-ID: <3BDD1519.31496FB@lemburg.com> [Integrating more crypto code into Python] Please don't proceed in this direction. Crypto code has a long tradition of making world-wide distribution hard and painful, either to export, to import or to usage restrictions. There's a good reason why e.g. mxCrypto was developed outside the US, in fact I wrote that package because I couldn't legally export Andrew's code from the US. OpenSSL itself was developed in large parts in Australia for much the same reason. -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Consulting & Company: http://www.egenix.com/ Python Software: http://www.lemburg.com/python/ From mal@lemburg.com Mon Oct 29 08:49:22 2001 From: mal@lemburg.com (M.-A. Lemburg) Date: Mon, 29 Oct 2001 09:49:22 +0100 Subject: [Python-Dev] Adding DB modules to the core ? (Re: Future of SSL) References: <20011027014211.A11092@lilith.hqd-internal> <200110270226.WAA23767@cj20424-a.reston1.va.home.com> <20011026235527.A2919@trump.amber.org> <200110271556.LAA25312@cj20424-a.reston1.va.home.com> Message-ID: <3BDD1812.1FEF2E33@lemburg.com> Guido van Rossum wrote: > > (Note that, alas, the DB-API never made it this far -- we still > haven't been able to find the time to do the tedious work of moving > the various database adapters in the core distribution. :-( ) Do you really think that this would be the right approach ? Sounds like code bloat to me... BTW, I'm still waiting for one of the PSF sponsors to create the mythical Sumo distribution of Python. Any news in that direction ? -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Consulting & Company: http://www.egenix.com/ Python Software: http://www.lemburg.com/python/ From jack@oratrix.nl Mon Oct 29 10:02:04 2001 From: jack@oratrix.nl (Jack Jansen) Date: Mon, 29 Oct 2001 11:02:04 +0100 Subject: [Python-Dev] Re: patchlevel.h In-Reply-To: Message by Guido van Rossum <guido@python.org> , Fri, 26 Oct 2001 14:56:20 -0400 , <200110261856.OAA20424@cj20424-a.reston1.va.home.com> Message-ID: <20011029100204.E0766303181@snelboot.oratrix.nl> > > While this may happen in principle, and out of curiosity: Was this a > > problem in 2.2b1 also? Looking at the changes made on the 2.2b1 > > release branch, I see that a total of 7 files was changed. Except for > > patchlevel.h, and the \n\ fix on socketmodule.c, there were no changes > > to C code. > > What's more common is actually a flurry of checkins just *before* the > release branch is made (say in the last two days). These often > include a somewhat hasty commit of a project that's valuable to have > but introduces new bugs -- either Mac-specific or generic. The patch > that introduced the missing \n\ problem was an example of this. And, more serious, there's another example for the 2.2b1 release that I found just friday (so this bug is going to stay in MacPython 2.2b1, I'm afraid): importing packages from frozen programs is broken. There was a moderately simple fix to import.c, to test that submodules of frozen modules were also frozen, and not picked up from the filesystem. Unfortunately whoever did the patch wasn't aware that MacPython frozen modules work different, and hence broke it. -- Jack Jansen | ++++ stop the execution of Mumia Abu-Jamal ++++ Jack.Jansen@oratrix.com | ++++ if you agree copy these lines to your sig ++++ www.cwi.nl/~jack | see http://www.xs4all.nl/~tank/spg-l/sigaction.htm From jack@oratrix.nl Mon Oct 29 10:10:16 2001 From: jack@oratrix.nl (Jack Jansen) Date: Mon, 29 Oct 2001 11:10:16 +0100 Subject: [Python-Dev] Re: patchlevel.h In-Reply-To: Message by Guido van Rossum <guido@python.org> , Fri, 26 Oct 2001 14:56:20 -0400 , <200110261856.OAA20424@cj20424-a.reston1.va.home.com> Message-ID: <20011029101016.E80A6303181@snelboot.oratrix.nl> > > It's just that I think very strict principles must be applied for the > > final release. If that means that the 2.2 release can't go without ok > > from you (or somebody else who has produced MacOS 9 binaries), I think > > we should add that to the release procedure. That check would be to > > avoid show-stopper bugs only, of course - anything complex needs to be > > detected long before the final release. > > I think that for the final release we should definitely attempt this. > Hopefully the speed and quality of change will reduce closer to the > final release. Also, the release candidate should help. For the final release I agree that this is a good idea. Let's try and make it so. -- Jack Jansen | ++++ stop the execution of Mumia Abu-Jamal ++++ Jack.Jansen@oratrix.com | ++++ if you agree copy these lines to your sig ++++ www.cwi.nl/~jack | see http://www.xs4all.nl/~tank/spg-l/sigaction.htm From just@letterror.com Mon Oct 29 10:24:33 2001 From: just@letterror.com (Just van Rossum) Date: Mon, 29 Oct 2001 11:24:33 +0100 Subject: [Python-Dev] Re: patchlevel.h In-Reply-To: <20011029100204.E0766303181@snelboot.oratrix.nl> Message-ID: <20011029112439-r01010800-90687d33-0910-010c@10.0.0.23> Jack Jansen wrote: > And, more serious, there's another example for the 2.2b1 release that I found > just friday (so this bug is going to stay in MacPython 2.2b1, I'm afraid): > importing packages from frozen programs is broken. > > There was a moderately simple fix to import.c, to test that submodules of > frozen modules were also frozen, and not picked up from the filesystem. > Unfortunately whoever did the patch wasn't aware that MacPython frozen modules > work different, and hence broke it. I remember seeing that, and meaning to look into it but forgot. I think it's a highly questionable fix, not only for MacPython: I'm not sure if this still allows C extensions as submodules in frozen apps. Just From chimaadams@yahoo.com Mon Oct 29 10:42:53 2001 From: chimaadams@yahoo.com (chima adams) Date: Mon, 29 Oct 2001 11:42:53 +0100 Subject: [Python-Dev] (no subject) Message-ID: <E15y9wC-0000a2-00@mail.python.org> FROM THE DESK:Dr.CHIMA ADAMS, ACCOUNT DEPARTMENT (NNPC) FALOMO OFFICE COMPLEX, IKOYI – LAGOS, NIGERIA TEL:234-8033084103 FAX: 234-1-7599186 ATTN: GENERAL MANAGER Dear Sir, I am Dr.Chima Adams chairman- audit committee, with the Nigerian National Petroleum Corporation (NNPC), I headed the audit committee whose assignment was to audit, review and recommend payment for all contracts awarded in the NNPC by the corrupt past military administration. In the course of our review, we discovered that the sum of USD 30 million (THIRTY MILLION DOLLARS) was floating unclaimed in the corporation account without beneficiary. This money emanated from an over-invoicing in the contract awarded to a foreign firm for the supply, fixing, erecting and computerized optimization of oil pipeline to one of our oil refineries during the past Military administration. Base on our status as senior government officials in top sensitive position of trust, we cannot stand to claim this money ourselves, hence, I decided to contact you. All modalities for this transfer have been fully worked out. We have already secured a preliminary order to effect this payment, we now require from you your BANKING PARTICULARS, FAX AND TELEPHONE NUMBER for smooth transfer of this money into your bank account. We intend to share the money in the following manner: 30% for you, 65% for us and 5% for any miscellaneous expenses. This business is 100% risk-free. We look forward to a fruitful business relationship with you. Best Regards Dr. Chima Adams Account Department (NNPC) _________________________________________________________ Do You Yahoo!? Get your free @yahoo.com address at http://mail.yahoo.com From fredrik@pythonware.com Mon Oct 29 11:28:32 2001 From: fredrik@pythonware.com (Fredrik Lundh) Date: Mon, 29 Oct 2001 12:28:32 +0100 Subject: [Python-Dev] (no subject) References: <E15y9wC-0000a2-00@mail.python.org> Message-ID: <005f01c1606c$d8dc1140$ced241d5@hagrid> > In the course of our review, we discovered that the sum of > USD 30 million (THIRTY MILLION DOLLARS) was floating un- > claimed in the corporation account without beneficiary. > We intend to share the money in the following manner: 30% > for you, 65% for us and 5% for any miscellaneous expenses. $9 million to the PSF? not bad. maybe we should hand them over to VA Linux, to make sure sourceforge stays up another year or two. </F> From gerhard@bigfoot.de Mon Oct 29 12:36:40 2001 From: gerhard@bigfoot.de (Gerhard =?iso-8859-1?Q?H=E4ring?=) Date: Mon, 29 Oct 2001 13:36:40 +0100 Subject: [Python-Dev] Future of SSL In-Reply-To: <3BDD1519.31496FB@lemburg.com>; from mal@lemburg.com on Mon, Oct 29, 2001 at 09:36:41AM +0100 References: <20011027014211.A11092@lilith.hqd-internal> <200110270226.WAA23767@cj20424-a.reston1.va.home.com> <20011026235527.A2919@trump.amber.org> <200110271556.LAA25312@cj20424-a.reston1.va.home.com> <20011027214357.B6993@trump.amber.org> <3BDB6742.E143BF8B@zope.com> <20011028184024.A1066@lilith.hqd-internal> <3BDD1519.31496FB@lemburg.com> Message-ID: <20011029133640.A1804@lilith.hqd-internal> On Mon, Oct 29, 2001 at 09:36:41AM +0100, M.-A. Lemburg wrote: > [Integrating more crypto code into Python] > > Please don't proceed in this direction. Crypto code has a long > tradition of making world-wide distribution hard and painful, either > to export, to import or to usage restrictions. True. But there's not much you can do about stupid governments, except not voting for them next time. > There's a good reason why e.g. mxCrypto was developed outside the US, > in fact I wrote that package because I couldn't legally export > Andrew's code from the US. OpenSSL itself was developed in large parts > in Australia for much the same reason. The problem is that there needs to be some crypto support so that HTTPS, SMTP over TLS, etc. works if the crypto is available. So, at least the crypto hooks (SSL API) must be there, right? What would be your suggestions? Would you prefer to go in the direction of my original proposal - only providing a SSL API, but not the implementation? Gerhard -- mail: gerhard <at> bigfoot <dot> de registered Linux user #64239 web: http://www.cs.fhm.edu/~ifw00065/ OpenPGP public key id 86AB43C0 public key fingerprint: DEC1 1D02 5743 1159 CD20 A4B6 7B22 6575 86AB 43C0 reduce(lambda x,y:x+y,map(lambda x:chr(ord(x)^42),tuple('zS^BED\nX_FOY\x0b'))) From guido@python.org Mon Oct 29 12:54:22 2001 From: guido@python.org (Guido van Rossum) Date: Mon, 29 Oct 2001 07:54:22 -0500 Subject: [Python-Dev] Adding DB modules to the core ? (Re: Future of SSL) In-Reply-To: Your message of "Mon, 29 Oct 2001 09:49:22 +0100." <3BDD1812.1FEF2E33@lemburg.com> References: <20011027014211.A11092@lilith.hqd-internal> <200110270226.WAA23767@cj20424-a.reston1.va.home.com> <20011026235527.A2919@trump.amber.org> <200110271556.LAA25312@cj20424-a.reston1.va.home.com> <3BDD1812.1FEF2E33@lemburg.com> Message-ID: <200110291254.HAA12811@cj20424-a.reston1.va.home.com> > Guido van Rossum wrote: > > > > (Note that, alas, the DB-API never made it this far -- we still > > haven't been able to find the time to do the tedious work of moving > > the various database adapters in the core distribution. :-( ) > > Do you really think that this would be the right approach ? > Sounds like code bloat to me... It depends. This is definitely a frequently requested feature, and that's what I go by... Plus, I know that the various DB packages (mxODBC notwithstanding :-) are of, eh, varying quality. > BTW, I'm still waiting for one of the PSF sponsors to create > the mythical Sumo distribution of Python. Any news in that > direction ? I don't recall either of the sponsors promising, although I would say that ActivePython might be a good start. I remember one of the regular members promising this, but apparently his fork was larger than his mouth (as we say in Holland). --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Mon Oct 29 12:56:15 2001 From: guido@python.org (Guido van Rossum) Date: Mon, 29 Oct 2001 07:56:15 -0500 Subject: [Python-Dev] Re: patchlevel.h In-Reply-To: Your message of "Mon, 29 Oct 2001 11:02:04 +0100." <20011029100204.E0766303181@snelboot.oratrix.nl> References: <20011029100204.E0766303181@snelboot.oratrix.nl> Message-ID: <200110291256.HAA12835@cj20424-a.reston1.va.home.com> > There was a moderately simple fix to import.c, to test that > submodules of frozen modules were also frozen, and not picked up > from the filesystem. Unfortunately whoever did the patch wasn't > aware that MacPython frozen modules work different, and hence broke > it. Guilty as charged. I didn't write it but approved it and checked it in. But that's what beta testing is for... --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Mon Oct 29 13:06:28 2001 From: guido@python.org (Guido van Rossum) Date: Mon, 29 Oct 2001 08:06:28 -0500 Subject: [Python-Dev] Future of SSL In-Reply-To: Your message of "Mon, 29 Oct 2001 13:36:40 +0100." <20011029133640.A1804@lilith.hqd-internal> References: <20011027014211.A11092@lilith.hqd-internal> <200110270226.WAA23767@cj20424-a.reston1.va.home.com> <20011026235527.A2919@trump.amber.org> <200110271556.LAA25312@cj20424-a.reston1.va.home.com> <20011027214357.B6993@trump.amber.org> <3BDB6742.E143BF8B@zope.com> <20011028184024.A1066@lilith.hqd-internal> <3BDD1519.31496FB@lemburg.com> <20011029133640.A1804@lilith.hqd-internal> Message-ID: <200110291306.IAA12932@cj20424-a.reston1.va.home.com> > True. But there's not much you can do about stupid governments, except > not voting for them next time. Doesn't seem to help in the US. :-( > > There's a good reason why e.g. mxCrypto was developed outside the US, > > in fact I wrote that package because I couldn't legally export > > Andrew's code from the US. OpenSSL itself was developed in large parts > > in Australia for much the same reason. > > The problem is that there needs to be some crypto support so that HTTPS, > SMTP over TLS, etc. works if the crypto is available. So, at least the > crypto hooks (SSL API) must be there, right? > > What would be your suggestions? Would you prefer to go in the direction > of my original proposal - only providing a SSL API, but not the > implementation? Yes, that's how the current SSL support works -- you need to link in openssl. --Guido van Rossum (home page: http://www.python.org/~guido/) From gerhard@bigfoot.de Mon Oct 29 13:18:45 2001 From: gerhard@bigfoot.de (Gerhard =?iso-8859-1?Q?H=E4ring?=) Date: Mon, 29 Oct 2001 14:18:45 +0100 Subject: [Python-Dev] Future of SSL In-Reply-To: <200110291306.IAA12932@cj20424-a.reston1.va.home.com>; from guido@python.org on Mon, Oct 29, 2001 at 08:06:28AM -0500 References: <20011027014211.A11092@lilith.hqd-internal> <200110270226.WAA23767@cj20424-a.reston1.va.home.com> <20011026235527.A2919@trump.amber.org> <200110271556.LAA25312@cj20424-a.reston1.va.home.com> <20011027214357.B6993@trump.amber.org> <3BDB6742.E143BF8B@zope.com> <20011028184024.A1066@lilith.hqd-internal> <3BDD1519.31496FB@lemburg.com> <20011029133640.A1804@lilith.hqd-internal> <200110291306.IAA12932@cj20424-a.reston1.va.home.com> Message-ID: <20011029141844.A1967@lilith.hqd-internal> On Mon, Oct 29, 2001 at 08:06:28AM -0500, Guido van Rossum wrote: > > [me, in response to a remark from Marc-André] > > What would be your suggestions? Would you prefer to go in the direction > > of my original proposal - only providing a SSL API, but not the > > implementation? > > Yes, that's how the current SSL support works -- you need to link in > openssl. So, as long as there are no actual cryptographic algorithms in the Python source tree, but only hooks for OpenSSL, there's no problem? If this is the case, then there would only be an issue for binary distributions of Python, which can be built easily in free countries :-) Gerhard -- mail: gerhard <at> bigfoot <dot> de registered Linux user #64239 web: http://www.cs.fhm.edu/~ifw00065/ OpenPGP public key id 86AB43C0 public key fingerprint: DEC1 1D02 5743 1159 CD20 A4B6 7B22 6575 86AB 43C0 reduce(lambda x,y:x+y,map(lambda x:chr(ord(x)^42),tuple('zS^BED\nX_FOY\x0b'))) From jack@oratrix.nl Mon Oct 29 14:07:14 2001 From: jack@oratrix.nl (Jack Jansen) Date: Mon, 29 Oct 2001 15:07:14 +0100 Subject: [Python-Dev] PEP 273 - inporting from zipfiles Message-ID: <20011029140714.81E46303181@snelboot.oratrix.nl> Jim, for your implementation you might want to have a look at the Macintosh-specific code in import.c (and possibly at related code in Mac/macglue.c, but that's probably less relevant). MacPython can already have files on sys.path, and there's code to look through these files for PYC resources. Don't worry what they are or how it's done, but they contain compiled modules. I got a pretty significant increase of import speed is when I added a cache of which sys.path entries were files and which were directories, so for each sys.path entry you know that you can skip one leg of the find_module() code. This is all in import.c, so you could probably enable it verbatim for zipfile imports. -- Jack Jansen | ++++ stop the execution of Mumia Abu-Jamal ++++ Jack.Jansen@oratrix.com | ++++ if you agree copy these lines to your sig ++++ www.cwi.nl/~jack | ++++ see http://www.xs4all.nl/~tank/ ++++ From jim@interet.com Mon Oct 29 14:19:52 2001 From: jim@interet.com (James C. Ahlstrom) Date: Mon, 29 Oct 2001 09:19:52 -0500 Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives References: <3BD9B5EC.119319A5@interet.com> <200110262034.QAA20802@cj20424-a.reston1.va.home.com> Message-ID: <3BDD6588.7CFEA7D6@interet.com> Guido van Rossum wrote: > Hm, I wonder if we couldn't just link with the libz.a C library and > use the C interface, if you're implementing this in C anyway. Since zlib is often in a DLL or shared library (.so), I am afraid of portability problems if we use the C interface. For example, on Windows, linking python.dll with zlib.dll would cause python.dll to fail if zlib.dll were unavailable, even if zlib.dll were never accessed. It also happens that use of the zlib module is easier. > I wonder if we need to make an additional rule that allows a .pyc file > to satisfy a module request even if we're in optimized mode (where > normally only .pyo files are searched). Otherwise, if someone ships a > zipfile with only .pyc files, their modules can't be imported at all > when python -O is used. Yes, I agree. How about if we look for the correct .py[co] first, and if that fails, look for the other? Either will satisfy the import, right? If .pyc is wanted, .pyo is OK too. > | The only way to find files in a zip archive is linear search. > > But there's an index record at the end that provides quick access. It is this index which is searched linearly. > Do we get an "module not found" error or something better, like > "compressed module found as <filename> but zlib unavailable"? We get "module not found". The second is awkward, because the module may be found later in sys.path. > Well, this is the domain of getpath.c, and that's got a different > implementation for Unix and Windows anyway (Windows has PC/getpathp.c). I would like discussion on what the additional sys.path names are. > | A C implementation exists which works, but which can be made better. I can upload it for inspection, but I don't want it to be a patch because it is not done yet. JimA From mal@lemburg.com Mon Oct 29 14:20:25 2001 From: mal@lemburg.com (M.-A. Lemburg) Date: Mon, 29 Oct 2001 15:20:25 +0100 Subject: [Python-Dev] Future of SSL References: <20011027014211.A11092@lilith.hqd-internal> <200110270226.WAA23767@cj20424-a.reston1.va.home.com> <20011026235527.A2919@trump.amber.org> <200110271556.LAA25312@cj20424-a.reston1.va.home.com> <20011027214357.B6993@trump.amber.org> <3BDB6742.E143BF8B@zope.com> <20011028184024.A1066@lilith.hqd-internal> <3BDD1519.31496FB@lemburg.com> <20011029133640.A1804@lilith.hqd-internal> Message-ID: <3BDD65A9.BB2988A3@lemburg.com> Gerhard H=E4ring wrote: >=20 > On Mon, Oct 29, 2001 at 09:36:41AM +0100, M.-A. Lemburg wrote: > > [Integrating more crypto code into Python] > > > > Please don't proceed in this direction. Crypto code has a long > > tradition of making world-wide distribution hard and painful, either > > to export, to import or to usage restrictions. >=20 > True. But there's not much you can do about stupid governments, except > not voting for them next time. I think it's more difficult than that... this is about politics and can be a very touchy subject depending on at which angle you look=20 at it. > > There's a good reason why e.g. mxCrypto was developed outside the US, > > in fact I wrote that package because I couldn't legally export > > Andrew's code from the US. OpenSSL itself was developed in large part= s > > in Australia for much the same reason. >=20 > The problem is that there needs to be some crypto support so that HTTPS= , > SMTP over TLS, etc. works if the crypto is available. So, at least the > crypto hooks (SSL API) must be there, right? >=20 > What would be your suggestions? Would you prefer to go in the direction > of my original proposal - only providing a SSL API, but not the > implementation? Note that even the hooks can cause legal problems, because some (countries) consider the hooks as guns without amunition. I'd suggest to leave serious crypto completely out of Python and to use one of the available third-party toolkits in case it should become necessary, e.g. M2Crypto has been around for a while and probably provides all that's necessary to do HTTPS or other SSL/TLS-based protocol and amkCrypto provides all the low-level tools to write your own secure procotols. --=20 Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Consulting & Company: http://www.egenix.com/ Python Software: http://www.lemburg.com/python/ From gmcm@hypernet.com Mon Oct 29 14:30:30 2001 From: gmcm@hypernet.com (Gordon McMillan) Date: Mon, 29 Oct 2001 09:30:30 -0500 Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives In-Reply-To: <3BD9B5EC.119319A5@interet.com> Message-ID: <3BDD21B6.12124.38E143B8@localhost> Jim Ahlstrom wrote: [From PEP 273] > Currently, sys.path is a list of directory names as strings. A nit: that's in practice, but not by definition (a couple std modules have been patched to allow other things on sys.path). After extensive use of imputil (which puts objects on sys.path), I think we might as well make it official that sys.path is a list of strings. [Subdirectory Equivalence] This is a bit misleading - it can be read as implying that the importable modules are in a flat namespace. They're not. To get R.Q.modfoo imported, R.__init__ and R.Q.__init__ must be imported. The __init__ modules have the opportunity to play with __path__ to break the equivalence of R.Q.modfoo to R/Q/modfoo.py. Or (more likely), play games with attributes so that Q is, say, an instance, or maybe a module imported from someplace else. Question: if Archive.zip/Q/__init__.pyc does "__path__.append('some/real/directory')", will it work? [dynamic libs] > It might be possible to extract the dynamic module from the > zip file, write it to a plain file and load it. I think you can nail the door shut on this one. On many OSes, making dynamic libs available to a process requires settings that can only (sanely) be made before the process starts. OTOH, it's common practice these days to put dynamic libs inside packages. That needs to be dealt with at runtime, and at build time (since it breaks the expectation that you can just "zip up" the package). [no *.py files] > You also can't import source files *.py from a zip archive. Apparently some Linuxes / RPM distributions don't deliver .pyc's or .pyo's. Since they're installed as root and run as some-poor-user, I'm afraid there are quite a few installations running off .py's all the time. So while it's definitely sub- optimal, I'm not sure it should be outlawed. [Guido] > But it would still be good if the .py files were also in the > zip file, so the source can be used in tracebacks etc. As you probably remember me saying before, I think this is a very minor nice-to-have. For one thing, you've probably done enough testing before stuffing your code into an archive that you're past the point of looking at a traceback and going "doh!". Most of the time you'll need to look at the code anyway. OTOH, this [more from Guido]: > A C API to get a source line from a filename might be a good > idea (plus a Python API). points the way towards something I'm very much in favor of: deferring things to that-which-did-the-importing. [Efficiency] > The key is the archive name from sys.path joined with the > file name (including any subdirectories) within the archive. DIfferent spellings of the same path are possible in a filesystem, but not in a dictionary. A bit of "harmless" tweaking of sys.path could render an archive unreachable. [zlib must be available at start] I'll agree, and agree with Guido that the coolest thing would be to make zlib standard. [Booting - python.zip should be part of the generated sys.path] Agree. Nice and straightforward. Now, from the discussion: [restate Jim] sys.path contains /C/D/E/Archive.zip and Archive.zip contains "Q/R/modfoo.pyc" so "import Q.R.modfoo" is satisfied by /C/D/E/Archive.zip/Q/R/modfoo.pyc [restate Finn] Jython has sys.path /C/D/E/Archive.zip!Lib and Archive.zip has "Lib/Q/R/modfoo.pyc" so "import Q.R.modfoo" is satisfied by /C/D/E/Archive.zip/Lib/Q/R/modfoo.pyc [restate Guido] Why not use /C/D/E/Archive.zip/Lib on sys.path? I use embedded archives. So sys.path will have an entry like: "/path/to/executable?84758" which says that the archive starts at position 84758 in the file "executable". Anything beyond the simple approach Jim takes gets into some very URL-ish territory. That's fine by me :-). I don't really like the idea of hacking special knowledge of zip files into import.c (which is already a specialist in filesystems). Like Finn said, if this is a deployment issue (we want zip files now, and are willing to live with strict limitations / rules to get it), then OK (as long as it supports __path__ and some way of dealing with dynamic libs in packages). Personally, I think package support stretched import.c to it's monolithic limits and it's high time the code was refactored to make it sanely extensible. - Gordon From jim@interet.com Mon Oct 29 14:35:07 2001 From: jim@interet.com (James C. Ahlstrom) Date: Mon, 29 Oct 2001 09:35:07 -0500 Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives References: <3BD9B5EC.119319A5@interet.com> <3bd9d419.10386565@mail.wanadoo.dk> Message-ID: <3BDD691B.2599AAF4@interet.com> Finn Bock wrote: > Why can't .py be allowed? If a more recent .py[co] (or $py.class) > exists, it is used. Otherwise the .py file is compiled and discarded > when the process ends. Sure, it is slower, but a zip files with only > .py[co] entries would be of little use with jython. The *.py are allowed to be in the file, and jpython can use them. In fact, any files at all can be in the archive. It is just that C-Python ignores them. My reason was that if *.py[co] are missing or out of date, zip importing will be slow and users won't figure out what is wrong. But I am open to changing this. > > Just as sys.path currently has default directory names, default > A standard for this would be really cool. Yes, lets make a standard now. > So python packages and modules can exists *only* at the top level? That > would conflict somewhat with jython where at would be common to put > python modules into the same .zip file as java classes and java classes > also wants to own the root of a zip file. > > In the current implementaion in jython, we can put python modules under > any path and add the zipfile!path to sys.path: > > sys.path.append("/path/to/zipfile.zip!Lib") > > which will look for Lib/Q/R/modfoo.py (and Lib/Q/R/modfoo$py.class) in > the archive. I am confused. Zip archives are equivalant to subdirectories, so there is no requirement to have anything at the top level. Your example seems to imply a second search path besides sys.path. BTW, the code uses ".zip" as the archive flag, not a special character '!'. > [I found efficiency hard to achieve in jython because opening a zipfile > in java also cause the zip index to be read into a dictionary. So we > did not want to reopen a zipfile if it can be avoided. Instead we hide a > reference to the opened file in sys.path so that when removing a zipfile > name from sys.path, the file is eventually closed.] > > Would entries in the static python dict be removed when a zipfile is > removed from sys.path? No, entries would not be removed. But they would not be found either, because their names would not be generated from the new sys.path. > What is the __path__ vrbl set to in a module imported from a zipfile? > Can the module make changes to __path__ and will be changes to used when > importing submodules? > > What value should __file__ have? The __file__ is /A/B/archive.zip/name.py. There is no special code for __file__ nor __path__, the path name just has a ".zip" in it. JimA From guido@python.org Mon Oct 29 14:34:58 2001 From: guido@python.org (Guido van Rossum) Date: Mon, 29 Oct 2001 09:34:58 -0500 Subject: [Python-Dev] Future of SSL In-Reply-To: Your message of "Mon, 29 Oct 2001 14:18:45 +0100." <20011029141844.A1967@lilith.hqd-internal> References: <20011027014211.A11092@lilith.hqd-internal> <200110270226.WAA23767@cj20424-a.reston1.va.home.com> <20011026235527.A2919@trump.amber.org> <200110271556.LAA25312@cj20424-a.reston1.va.home.com> <20011027214357.B6993@trump.amber.org> <3BDB6742.E143BF8B@zope.com> <20011028184024.A1066@lilith.hqd-internal> <3BDD1519.31496FB@lemburg.com> <20011029133640.A1804@lilith.hqd-internal> <200110291306.IAA12932@cj20424-a.reston1.va.home.com> <20011029141844.A1967@lilith.hqd-internal> Message-ID: <200110291434.JAA13268@cj20424-a.reston1.va.home.com> > > > [me, in response to a remark from Marc-André] > > > What would be your suggestions? Would you prefer to go in the direction > > > of my original proposal - only providing a SSL API, but not the > > > implementation? > > > > Yes, that's how the current SSL support works -- you need to link in > > openssl. > > So, as long as there are no actual cryptographic algorithms in the > Python source tree, but only hooks for OpenSSL, there's no problem? That's what I believe the situation currently, for open source software. It wasn't always like this -- at some point, hooks were enough to need export permission. It may again be like this, if some uninformed US senators get their way... > If this is the case, then there would only be an issue for binary > distributions of Python, which can be built easily in free countries :-) Hm, but that's dangerous too -- someone could easily build an RPM that includes openssl without realizing it. --Guido van Rossum (home page: http://www.python.org/~guido/) From jim@interet.com Mon Oct 29 14:45:08 2001 From: jim@interet.com (James C. Ahlstrom) Date: Mon, 29 Oct 2001 09:45:08 -0500 Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives References: <3BD9B5EC.119319A5@interet.com> <3bd9d419.10386565@mail.wanadoo.dk> <200110270223.WAA23737@cj20424-a.reston1.va.home.com> <3bdbfda4.3697676@mail.wanadoo.dk> Message-ID: <3BDD6B74.D4E19953@interet.com> Finn Bock wrote: > >> Would entries in the static python dict be removed when a zipfile is > >> removed from sys.path? > > > >It can be arranged that they are removed at some later point > >(e.g. when sys.path is next searched). > > An API to do this could be usefull for jython. Right now we depend on > the GC thread to close the file. Since files are a limited resource it > would be a good thing to have an explicit way to clean up the cached > resources. The static Python dictionary is a memory object which uses no open file descriptors. If an element of sys.path contains ".zip" and hasn't been seen before, the zip archive is opened, searched, and closed. The key is the name, the value includes the archive name and offset. Thereafter, the zip archive is never opened unless it contains a needed file. So I don't think there is a problem. > myself), I just didn't dare to put such a limit on my jython > implementation. I don't understand the jpython implementation, so please point out all problems so we can fix them now. JimA From gmcm@hypernet.com Mon Oct 29 14:50:30 2001 From: gmcm@hypernet.com (Gordon McMillan) Date: Mon, 29 Oct 2001 09:50:30 -0500 Subject: [Python-Dev] Future of SSL In-Reply-To: <200110291434.JAA13268@cj20424-a.reston1.va.home.com> References: Your message of "Mon, 29 Oct 2001 14:18:45 +0100." <20011029141844.A1967@lilith.hqd-internal> Message-ID: <3BDD2666.4196.38F3926B@localhost> I wrote some proprietary code for a client, using (hacked) M2Crypto and openSSL, but using only fairly lightweight DES (not even DES3). In order to get an export license, I had to talk to the Commerce Dept and the NSA. That got the export license, but to get it into France, the client had to talk extensively to the French government. My guess is that anything beyond enabling https in core Python will make negotiating with the FSF look like a Mardi Gras party. - Gordon From mmuller@enduden.com Mon Oct 29 14:35:40 2001 From: mmuller@enduden.com (Michael Muller) Date: Mon, 29 Oct 2001 09:35:40 -0500 Subject: [Python-Dev] status of OS/2 ports of Python Message-ID: <193.1004366140.804197.425672245@lilbastard> Guido van Rossum wrote: > > I would also propose to look after patches to the VAC++ build (such as > > Michael Muller's), on the basis that although I can't test the patches > > directly, I can test that they don't affect the rest of the system, and I > > can work with the submitter to resolve issues that affect OS/2 specific > > functionality. > > That sounds like a very good idea. Right now I don't really know what > to do with the two sets of OS/2 patches... Since Michael is cc'ed > here, Michael, what do you think of this? This works for me. As I've said in the notes on my patches, there is some "textual" conflict between Andrews patches and mine (meaning that my patches won't work after his have been applied). However, I see no functional conflicts, and as I've said I'd be happy to create new patches that will work with his. Also, Andrew: I have a lot of experience with CVS and I'm willing to offer any help you need. I can also test changes on Linux. My only wish is that this be resolved quickly so I don't have to keep revising my patches to keep up with the (rapidly evolving) baseline ;-). ============================================================================= michaelMuller = mmuller@enduden.com | http://www.cloud9.net/~proteus ----------------------------------------------------------------------------- In this book it is spoken of the Sephiroth, and the Paths, of Spirits and Conjurations; of Gods, Spheres, Planes and many other things which may or may not exist. It is immaterial whether they exist or not. By doing certain things certain results follow. - Aleister Crowley ============================================================================= From jim@interet.com Mon Oct 29 14:53:26 2001 From: jim@interet.com (James C. Ahlstrom) Date: Mon, 29 Oct 2001 09:53:26 -0500 Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives References: <3BD9B5EC.119319A5@interet.com> <3BDB1EB0.C2A9B1FF@lemburg.com> Message-ID: <3BDD6D66.675A4F3C@interet.com> "M.-A. Lemburg" wrote: > 1. Why are .py files being outlawed ? My reason was that if *.py[co] are missing or out of date, zip importing will be slow and users won't figure out what is wrong. Generally I favor user-proof features over expert features. I prefer things which either "Just Work" or fail spectacularly. But I am open to changing this. > 2. Where's the C implementation you mention in the PEP ? Software is like pancakes, you should always throw the first one away. I will post it if you want, but it is not done. > 3. Would it be possible to ship zlib together with Python ? > (the zlib license should allow this and I don't think that > the code size is too big) OK by me. But uncompressed zip archives work, and may even be faster than conpressed archives. JimA From guido@python.org Mon Oct 29 14:58:39 2001 From: guido@python.org (Guido van Rossum) Date: Mon, 29 Oct 2001 09:58:39 -0500 Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives In-Reply-To: Your message of "Mon, 29 Oct 2001 09:19:52 EST." <3BDD6588.7CFEA7D6@interet.com> References: <3BD9B5EC.119319A5@interet.com> <200110262034.QAA20802@cj20424-a.reston1.va.home.com> <3BDD6588.7CFEA7D6@interet.com> Message-ID: <200110291458.JAA13956@cj20424-a.reston1.va.home.com> > Guido van Rossum wrote: > > > Hm, I wonder if we couldn't just link with the libz.a C library and > > use the C interface, if you're implementing this in C anyway. > > Since zlib is often in a DLL or shared library (.so), I am afraid of > portability problems if we use the C interface. For example, on Windows, > linking python.dll with zlib.dll would cause python.dll to fail if zlib.dll > were unavailable, even if zlib.dll were never accessed. It also happens > that use of the zlib module is easier. OK, good enough. (I'm getting more and more curious about your implementation. ;-) > > I wonder if we need to make an additional rule that allows a .pyc file > > to satisfy a module request even if we're in optimized mode (where > > normally only .pyo files are searched). Otherwise, if someone ships a > > zipfile with only .pyc files, their modules can't be imported at all > > when python -O is used. > > Yes, I agree. How about if we look for the correct .py[co] first, > and if that fails, look for the other? Either will satisfy the > import, right? If .pyc is wanted, .pyo is OK too. Sounds good to me -- this might be a good rule anyhoo. > > | The only way to find files in a zip archive is linear search. > > > > But there's an index record at the end that provides quick access. > > It is this index which is searched linearly. That can't possibly take any time compared to other stuff -- it's a relatively short in-memory table. > > Do we get an "module not found" error or something better, like > > "compressed module found as <filename> but zlib unavailable"? > > We get "module not found". The second is awkward, because the > module may be found later in sys.path. Hm. Does it at least spit out a warning when zlib is needed but not available? It can be awkward to debug the situation where you get a module from the filesystem rather than the version from the zipfile, even though it is in the zipfile. While the compression method is listed in the zip info, it's not the first thing someone would look for unless they were aware that this failure mode existed. Since obviously the intention of putting the module in the zipfile was that it should be found, I think that failure to decompress should be turned into an immediate error -- the same way as a syntax error gets reported and not turned into a "skip to the next directory in sys.path" effect. > > Well, this is the domain of getpath.c, and that's got a different > > implementation for Unix and Windows anyway (Windows has PC/getpathp.c). > > I would like discussion on what the additional sys.path names are. Well, propose some. > > | A C implementation exists which works, but which can be made better. > > I can upload it for inspection, but I don't want it to be a patch > because it is not done yet. Just say so in the patch. You can upoad anything you want. :-) --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Mon Oct 29 15:06:59 2001 From: guido@python.org (Guido van Rossum) Date: Mon, 29 Oct 2001 10:06:59 -0500 Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives In-Reply-To: Your message of "Mon, 29 Oct 2001 09:30:30 EST." <3BDD21B6.12124.38E143B8@localhost> References: <3BDD21B6.12124.38E143B8@localhost> Message-ID: <200110291506.KAA14000@cj20424-a.reston1.va.home.com> > Jim Ahlstrom wrote: > [From PEP 273] > > Currently, sys.path is a list of directory names as strings. [GMcM] > A nit: that's in practice, but not by definition (a couple std > modules have been patched to allow other things on > sys.path). After extensive use of imputil (which puts objects > on sys.path), I think we might as well make it official that > sys.path is a list of strings. Interesting. So you think imputil is wrong to put objects there? Why? (Not arguing, just interested in your experience.) > [Subdirectory Equivalence] > > This is a bit misleading - it can be read as implying that the > importable modules are in a flat namespace. They're not. To > get R.Q.modfoo imported, R.__init__ and R.Q.__init__ must > be imported. The __init__ modules have the opportunity to > play with __path__ to break the equivalence of R.Q.modfoo to > R/Q/modfoo.py. Or (more likely), play games with attributes > so that Q is, say, an instance, or maybe a module imported > from someplace else. > > Question: if Archive.zip/Q/__init__.pyc does > "__path__.append('some/real/directory')", will it work? It should. > [dynamic libs] > > It might be possible to extract the dynamic module from the > > zip file, write it to a plain file and load it. > > I think you can nail the door shut on this one. On many OSes, > making dynamic libs available to a process requires settings > that can only (sanely) be made before the process starts. ANd I believe some systems require shared libraries to be owned by root. > OTOH, it's common practice these days to put dynamic libs > inside packages. That needs to be dealt with at runtime, and > at build time (since it breaks the expectation that you can just > "zip up" the package). Yes. This is important. > [no *.py files] > > You also can't import source files *.py from a zip archive. > > Apparently some Linuxes / RPM distributions don't deliver > .pyc's or .pyo's. Since they're installed as root and run as > some-poor-user, I'm afraid there are quite a few installations > running off .py's all the time. So while it's definitely sub- > optimal, I'm not sure it should be outlawed. Heh, this is an argument for Jim's position -- it would have come out during testing that way, since the imports would fail. ;-) > [Guido] > > But it would still be good if the .py files were also in the > > zip file, so the source can be used in tracebacks etc. > > As you probably remember me saying before, I think this is a > very minor nice-to-have. For one thing, you've probably done > enough testing before stuffing your code into an archive that > you're past the point of looking at a traceback and going > "doh!". Most of the time you'll need to look at the code > anyway. Testing is for wimps. :-) > OTOH, this [more from Guido]: > > A C API to get a source line from a filename might be a good > > idea (plus a Python API). > > points the way towards something I'm very much in favor of: > deferring things to that-which-did-the-importing. Yup. > [Efficiency] > > The key is the archive name from sys.path joined with the > > file name (including any subdirectories) within the archive. > > DIfferent spellings of the same path are possible in a > filesystem, but not in a dictionary. A bit of "harmless" > tweaking of sys.path could render an archive unreachable. Hm, wouldn't the archive just be opened a second time? Or do I misundestand you? > [zlib must be available at start] > I'll agree, and agree with Guido that the coolest thing would be > to make zlib standard. But we'd have to make sure it's statically linked. (Fortunately, we already link it statically on Windows.) > [Booting - python.zip should be part of the generated sys.path] > Agree. Nice and straightforward. > > Now, from the discussion: > > [restate Jim] > sys.path contains /C/D/E/Archive.zip > and Archive.zip contains "Q/R/modfoo.pyc" > so "import Q.R.modfoo" is satisfied by > /C/D/E/Archive.zip/Q/R/modfoo.pyc > > [restate Finn] > Jython has sys.path /C/D/E/Archive.zip!Lib > and Archive.zip has "Lib/Q/R/modfoo.pyc" > so "import Q.R.modfoo" is satisfied by > /C/D/E/Archive.zip/Lib/Q/R/modfoo.pyc > > [restate Guido] > Why not use /C/D/E/Archive.zip/Lib on sys.path? > > I use embedded archives. So sys.path will have an entry like: > "/path/to/executable?84758" which says that the archive > starts at position 84758 in the file "executable". > > Anything beyond the simple approach Jim takes gets into > some very URL-ish territory. That's fine by me :-). > > I don't really like the idea of hacking special knowledge of zip > files into import.c (which is already a specialist in > filesystems). Like Finn said, if this is a deployment issue (we > want zip files now, and are willing to live with strict limitations / > rules to get it), then OK (as long as it supports __path__ and > some way of dealing with dynamic libs in packages). > > Personally, I think package support stretched import.c to it's > monolithic limits and it's high time the code was refactored to > make it sanely extensible. Yes -- this has been on my TODO list for ages. But who's gonna DO it? --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Mon Oct 29 15:13:14 2001 From: guido@python.org (Guido van Rossum) Date: Mon, 29 Oct 2001 10:13:14 -0500 Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives In-Reply-To: Your message of "Mon, 29 Oct 2001 09:35:07 EST." <3BDD691B.2599AAF4@interet.com> References: <3BD9B5EC.119319A5@interet.com> <3bd9d419.10386565@mail.wanadoo.dk> <3BDD691B.2599AAF4@interet.com> Message-ID: <200110291513.KAA14082@cj20424-a.reston1.va.home.com> [Finn] > > So python packages and modules can exists *only* at the top level? That > > would conflict somewhat with jython where at would be common to put > > python modules into the same .zip file as java classes and java classes > > also wants to own the root of a zip file. > > > > In the current implementaion in jython, we can put python modules under > > any path and add the zipfile!path to sys.path: > > > > sys.path.append("/path/to/zipfile.zip!Lib") > > > > which will look for Lib/Q/R/modfoo.py (and Lib/Q/R/modfoo$py.class) in > > the archive. [JimA] > I am confused. Zip archives are equivalant to subdirectories, so there > is no requirement to have anything at the top level. Your example seems > to imply a second search path besides sys.path. I think Finn simply means that the zipfile may have some redundant initial suffix to all filenames (e.g. "Lib/") which could be an artefact of how the zip file is created (zip up this directory) or intended as an aid for other uses of the zipfile, like unpacking. (In the tar world, it's considered impolite not to have a common prefix; without that, untarring can too easily populate an innocent user's home directory with thousands of untarred files.) > BTW, the code uses ".zip" as the archive flag, > not a special character '!'. That's cool. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Mon Oct 29 15:16:29 2001 From: guido@python.org (Guido van Rossum) Date: Mon, 29 Oct 2001 10:16:29 -0500 Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives In-Reply-To: Your message of "Mon, 29 Oct 2001 09:53:26 EST." <3BDD6D66.675A4F3C@interet.com> References: <3BD9B5EC.119319A5@interet.com> <3BDB1EB0.C2A9B1FF@lemburg.com> <3BDD6D66.675A4F3C@interet.com> Message-ID: <200110291516.KAA14105@cj20424-a.reston1.va.home.com> > > 3. Would it be possible to ship zlib together with Python ? > > (the zlib license should allow this and I don't think that > > the code size is too big) > > OK by me. But uncompressed zip archives work, and may even > be faster than conpressed archives. Good point. Uncompressed archives will definitely be faster. I'd be happy to legislate uncompressed archives, except I'm worried that not all tools commonly used to create zip archives make it easy to turn off compression. --Guido van Rossum (home page: http://www.python.org/~guido/) From jim@interet.com Mon Oct 29 15:34:30 2001 From: jim@interet.com (James C. Ahlstrom) Date: Mon, 29 Oct 2001 10:34:30 -0500 Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives References: <3BDD21B6.12124.38E143B8@localhost> Message-ID: <3BDD7706.A5BBE307@interet.com> Gordon McMillan wrote: > > > Currently, sys.path is a list of directory names as strings. > > A nit: that's in practice, but not by definition (a couple std > modules have been patched to allow other things on > sys.path). After extensive use of imputil (which puts objects > on sys.path), I think we might as well make it official that > sys.path is a list of strings. A nit on a nit: Non-strings can still be allowed, because I can ignore them. > [Subdirectory Equivalence] > > This is a bit misleading - it can be read as implying that the > importable modules are in a flat namespace. They're not. To > get R.Q.modfoo imported, R.__init__ and R.Q.__init__ must > be imported. The __init__ modules have the opportunity to > play with __path__ to break the equivalence of R.Q.modfoo to > R/Q/modfoo.py. Or (more likely), play games with attributes > so that Q is, say, an instance, or maybe a module imported > from someplace else. I feel that this is an excellent point, but I don't understand package import well enough to comment. > Question: if Archive.zip/Q/__init__.pyc does > "__path__.append('some/real/directory')", will it work? My guess (and it really is a guess) is Yes. The import.c code which generates directory names is untouched. My code simply looks for ".zip" in the generated name and branches into the zip archive at that point. So if the directory version works, the zip archive version should work too. > [no *.py files] > > You also can't import source files *.py from a zip archive. > > Apparently some Linuxes / RPM distributions don't deliver > .pyc's or .pyo's. Since they're installed as root and run as > some-poor-user, I'm afraid there are quite a few installations > running off .py's all the time. So while it's definitely sub- > optimal, I'm not sure it should be outlawed. For Linux/RPM's I think shipping a library directory is better than a zip archive. It is easier to hack on a directory. I think of zip archives as a way to distribute packages, and as a replacement for freeze. OTOH, maybe we should allow *.py to satisfy imports even if it is slow and invisible. > [Efficiency] > > The key is the archive name from sys.path joined with the > > file name (including any subdirectories) within the archive. > > DIfferent spellings of the same path are possible in a > filesystem, but not in a dictionary. A bit of "harmless" > tweaking of sys.path could render an archive unreachable. True, but I have little sympathy for case-insensitive file systems. Tweaking of sys.path will have to be done with care. It helps that the '/' character is always used in zip, and both backslash and colon ":" are illegal in zip archive names. JimA From jim@interet.com Mon Oct 29 15:38:33 2001 From: jim@interet.com (James C. Ahlstrom) Date: Mon, 29 Oct 2001 10:38:33 -0500 Subject: [Python-Dev] PEP 273 - inporting from zipfiles References: <20011029140714.81E46303181@snelboot.oratrix.nl> Message-ID: <3BDD77F9.949DA69D@interet.com> Jack Jansen wrote: > > Jim, > for your implementation you might want to have a look at the > Macintosh-specific code in import.c (and possibly at related code in > Mac/macglue.c, but that's probably less relevant). OK, will do. JimA From gmcm@hypernet.com Mon Oct 29 15:41:04 2001 From: gmcm@hypernet.com (Gordon McMillan) Date: Mon, 29 Oct 2001 10:41:04 -0500 Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives In-Reply-To: <200110291506.KAA14000@cj20424-a.reston1.va.home.com> References: Your message of "Mon, 29 Oct 2001 09:30:30 EST." <3BDD21B6.12124.38E143B8@localhost> Message-ID: <3BDD3240.31592.3921E147@localhost> [Gordon] > > After extensive use of imputil (which puts objects on > > sys.path), I think we might as well make it official that > > sys.path is a list of strings. [Guido] > Interesting. So you think imputil is wrong to put objects there? > Why? (Not arguing, just interested in your experience.) Too much code (some left in the std lib) says: for d in sys.path: os.path.join(d, ...) and that's the line that barfs. It doesn't barf if d is a string. [Gordon] > > DIfferent spellings of the same path are possible in a > > filesystem, but not in a dictionary. A bit of "harmless" > > tweaking of sys.path could render an archive unreachable. > > Hm, wouldn't the archive just be opened a second time? Or do I > misundestand you? Without seeing the C code, I don't know if it will open it a 2nd time. I keep an open file for each archive, but I'm careful that however it's spelled, I only open the archive once. Jim is apparently closing the file, so opening a 2nd is probably not as painful. OTOH an open / seek for each satisfied import is relatively expensive (though still cheaper than a fs import). > > [zlib must be available at start] > > I'll agree, and agree with Guido that the coolest thing would > > be to make zlib standard. > > But we'd have to make sure it's statically linked. (Fortunately, > we already link it statically on Windows.) So to avoid statically linking it twice, I assume zlib would be a mandatory builtin. [Gordon] > > Personally, I think package support stretched import.c to it's > > monolithic limits and it's high time the code was refactored to > > make it sanely extensible. > > Yes -- this has been on my TODO list for ages. But who's gonna > DO it? I published iu4 which I think comes very close to the "right" model. If it's close enough, I'll download 2.2 and see if I can make those into new style object subclasses[1]. - Gordon [1] As an app developer, I usually have to look backwards, not forwards. I've got 6 machines, each with at least 2 Pythons, so I no longer deal with betas :-(. Even when they have cool things like generators and subclassable C types. From barry@zope.com Mon Oct 29 15:52:27 2001 From: barry@zope.com (Barry A. Warsaw) Date: Mon, 29 Oct 2001 10:52:27 -0500 Subject: [Python-Dev] Re: patchlevel.h References: <20011026102540.3DBEB303181@snelboot.oratrix.nl> <200110261831.f9QIVob01347@mira.informatik.hu-berlin.de> <200110261856.OAA20424@cj20424-a.reston1.va.home.com> Message-ID: <15325.31547.431457.787791@anthem.wooz.org> >>>>> "GvR" == Guido van Rossum <guido@python.org> writes: GvR> I think that for the final release we should definitely GvR> attempt this. Hopefully the speed and quality of change will GvR> reduce closer to the final release. Also, the release GvR> candidate should help. GvR> Jack, We'll try to work with you. Barry, please add GvR> something to this effect in the release procedure PEP (hold GvR> up the *final* release until Jack approves or until we lose GvR> patience). Absolutely. And done. -Barry From gmcm@hypernet.com Mon Oct 29 15:54:50 2001 From: gmcm@hypernet.com (Gordon McMillan) Date: Mon, 29 Oct 2001 10:54:50 -0500 Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives In-Reply-To: <200110291516.KAA14105@cj20424-a.reston1.va.home.com> References: Your message of "Mon, 29 Oct 2001 09:53:26 EST." <3BDD6D66.675A4F3C@interet.com> Message-ID: <3BDD357A.126.392E7B98@localhost> [Guido] > Good point. Uncompressed archives will definitely be faster. > I'd be happy to legislate uncompressed archives, except I'm > worried that not all tools commonly used to create zip archives > make it easy to turn off compression. Don't be so sure about speed. If the archive is big enough and the OS bad enough <wink>, it may be faster to uncompress than swap. I always compress at 9, and have only seen speed ups of 1 or 2% with compression turned off. BTW, .pyc's compress quite well. - Gordon From jim@interet.com Mon Oct 29 15:58:10 2001 From: jim@interet.com (James C. Ahlstrom) Date: Mon, 29 Oct 2001 10:58:10 -0500 Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives References: <3BD9B5EC.119319A5@interet.com> <200110262034.QAA20802@cj20424-a.reston1.va.home.com> <3BDD6588.7CFEA7D6@interet.com> <200110291458.JAA13956@cj20424-a.reston1.va.home.com> Message-ID: <3BDD7C92.F130E708@interet.com> Guido van Rossum wrote: > Since obviously the intention of putting the module in the zipfile was > that it should be found, I think that failure to decompress should be > turned into an immediate error -- the same way as a syntax error gets > reported and not turned into a "skip to the next directory in > sys.path" effect. Hmmm, I'm not sure. But it is easy to do. > > I can upload it for inspection, but I don't want it to be a patch > > because it is not done yet. > > Just say so in the patch. You can upoad anything you want. :-) OK, I will post the implementation. JimA From gmcm@hypernet.com Mon Oct 29 16:17:50 2001 From: gmcm@hypernet.com (Gordon McMillan) Date: Mon, 29 Oct 2001 11:17:50 -0500 Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives In-Reply-To: <3BDD7706.A5BBE307@interet.com> Message-ID: <3BDD3ADE.19659.39438820@localhost> > Gordon McMillan wrote: > > [no *.py files] > > > You also can't import source files *.py from a zip archive. > > > > Apparently some Linuxes / RPM distributions don't deliver > > .pyc's or .pyo's. Since they're installed as root and run as > > some-poor-user, I'm afraid there are quite a few installations > > running off .py's all the time. So while it's definitely sub- > > optimal, I'm not sure it should be outlawed. > > For Linux/RPM's I think shipping a library directory is better > than a zip archive. It is easier to hack on a directory. I > think of zip archives as a way to distribute packages, and as a > replacement for freeze. There's no reason to believe that telling a Linux distribution maintainer that they "shouldn't" do it that way will be successful. Heck, he might be a Perl monkey on KP <wink>. > OTOH, maybe we should allow *.py to satisfy imports even if > it is slow and invisible. > > > [Efficiency] > > > The key is the archive name from sys.path joined with the > > > file name (including any subdirectories) within the archive. > > > > DIfferent spellings of the same path are possible in a > > filesystem, but not in a dictionary. A bit of "harmless" > > tweaking of sys.path could render an archive unreachable. > > True, but I have little sympathy for case-insensitive file > systems. Tweaking of sys.path will have to be done with care. It > helps that the '/' character is always used in zip, and both > backslash and colon ":" are illegal in zip archive names. You don't need case-insensitive for this to come up. Relative paths vs. absolute paths; or paths that need norm-ing. And then you've got the mac which uses a different path syntax, and even slightly different path semantics. - Gordon From jim@interet.com Mon Oct 29 16:31:42 2001 From: jim@interet.com (James C. Ahlstrom) Date: Mon, 29 Oct 2001 11:31:42 -0500 Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives References: <3BD9B5EC.119319A5@interet.com> <200110262034.QAA20802@cj20424-a.reston1.va.home.com> <3BDD6588.7CFEA7D6@interet.com> <200110291458.JAA13956@cj20424-a.reston1.va.home.com> Message-ID: <3BDD846E.7D372942@interet.com> Guido van Rossum wrote: > Just say so in the patch. You can upoad anything you want. :-) The implementation is in patch 476047. http://sourceforge.net/tracker/index.php?func=detail&aid=476047&group_id=5470&atid=305470 JimA From jim@interet.com Mon Oct 29 17:28:44 2001 From: jim@interet.com (James C. Ahlstrom) Date: Mon, 29 Oct 2001 12:28:44 -0500 Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives References: <3BD9B5EC.119319A5@interet.com> <200110262034.QAA20802@cj20424-a.reston1.va.home.com> <3BDD6588.7CFEA7D6@interet.com> <200110291458.JAA13956@cj20424-a.reston1.va.home.com> Message-ID: <3BDD91CC.9F0ECB07@interet.com> Guido van Rossum wrote: [Me] > > I would like discussion on what the additional sys.path names are. > > Well, propose some. OK. I propose that there is one name added to sys.path, and the file name is "python%s%s.zip" % (sys.version[0], sys.version[2]). For example, python22.zip. This is the same on all platforms. On Unix, the directory is sys.prefix + "/lib". So for prefix /usr/local, the path /usr/local/lib/python2.2/ is already on sys.path, and /usr/local/lib/python22.zip would be added. On Windows, the directory is the directory of sys.executable. The zip archive name is always inserted as the second item in sys.path. The first always seems to be ''. JimA From guido@python.org Mon Oct 29 17:39:37 2001 From: guido@python.org (Guido van Rossum) Date: Mon, 29 Oct 2001 12:39:37 -0500 Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives In-Reply-To: Your message of "Mon, 29 Oct 2001 12:28:44 EST." <3BDD91CC.9F0ECB07@interet.com> References: <3BD9B5EC.119319A5@interet.com> <200110262034.QAA20802@cj20424-a.reston1.va.home.com> <3BDD6588.7CFEA7D6@interet.com> <200110291458.JAA13956@cj20424-a.reston1.va.home.com> <3BDD91CC.9F0ECB07@interet.com> Message-ID: <200110291739.MAA15773@cj20424-a.reston1.va.home.com> > OK. I propose that there is one name added to sys.path, and the > file name is "python%s%s.zip" % (sys.version[0], sys.version[2]). > For example, python22.zip. This is the same on all platforms. > > On Unix, the directory is sys.prefix + "/lib". So for prefix > /usr/local, the path /usr/local/lib/python2.2/ is already on > sys.path, and /usr/local/lib/python22.zip would be added. > > On Windows, the directory is the directory of sys.executable. > > The zip archive name is always inserted as the second item > in sys.path. The first always seems to be ''. Sounds good to me. Somebody please update the PEP. --Guido van Rossum (home page: http://www.python.org/~guido/) From ngps@post1.com Mon Oct 29 17:52:01 2001 From: ngps@post1.com (Ng Pheng Siong) Date: Tue, 30 Oct 2001 01:52:01 +0800 Subject: [Python-Dev] Re: Python's SSL: Would you like to help? In-Reply-To: <20011029135307.C1844@lilith.hqd-internal>; from gerhard@bigfoot.de on Mon, Oct 29, 2001 at 01:53:07PM +0100 References: <20011029135307.C1844@lilith.hqd-internal> Message-ID: <20011030015201.B1007@madcap.dyndns.org> On Mon, Oct 29, 2001 at 01:53:07PM +0100, Gerhard Häring wrote: > I've meanwhile started a discussion about the future of SSL support in > Python itself on the python-dev mailinglist. > > The talk is about several issues, like redesign a whole new API > (yes/no), integrate an existing module (which one?), support much crypto > at all. > > As you seem to have a lot of experience in this field, perhaps you'd > like to share some of your opinions there? [ Not subscribed, hopefully this gets thru. ] Hello, Thanks for asking. I've just read the discussion in the archives. My own archives contain a similar (but much shorter) discussion that took place on the old python-crypto@egroups list in Apr 2000. I've also just downloaded 2.2b1, POW (Python OpenSSL wrappers) 0.5, and pyOpenSSL 0.4 to have a quick look-see. (Haven't been keeping track of the competition. ;-) I noticed there are several SSL patches on SF, mostly from Gerhard. I didn't read the patches, though. Here are some random thoughts based on my experience with M2Crypto: - To do simple multi-threading, you need to bracket calls to SSL_read(), SSL_write(), etc. with Py_BEGIN_ALLOW_THREADS and Py_END_ALLOW_THREADS. You also (apparently) need to implement the locking required by OpenSSL; see M2Crypto's _thread.i. IIRC, OpenSSL's docu warns of Bad Things should one fail to do so. (None of the others seem to do this.) If you want to do callbacks for certificate verification, it's a little more complicated. First, you need to muck around with Python's thread states in OpenSSL's structures. M2Crypto and pyOpenSSL do these mostly the same way. POW doesn't do this. Next, consider the Python "SSL Context" object that mirrors SSL_CTX: In M2Crypto, the "SSL Context" is implemented in Python; conveniently, I use it to keep additional Python-level policy stuff. Thus, when OpenSSL calls back into M2Crypto, I need a means to get at the Python SSL Context object, before calling back into Python. Implementing the SSL Context object in C may make this simpler. (I see 2.2b1 does only SSL_VERIFY_NONE. Ditto POW and pyOpenSSL, it seems.) - Even if you only want to do certificates within SSL, you need to wrap quite a bit of the X.509 API. See M2Crypto's _x509.i and X509.py. POW seems to have done so, as well. If you also want to do X.509 independent of SSL (say for implementing a CA), you'll need to do more housekeeping, like whether to free a given structure at the Python side or the C side. Grep for "_pyfree" in M2Crypto's Python code. - Asynchronous SSL: I think M2Crypto has this mostly right, since ZServerSSL works (2nd birthday came and went it seems ;-), and I've recently implemented Medusa-based FTP-over-TLS which interoperates with other people's clients. Based on my light reading of the code, none of 2.2b1, POW and pyOpenSSL do async-SSL properly at all. - FakeSocket, SSLFakeSocket and SSLFakeFile are ugly! M2Crypto wraps OpenSSL's "stackable BIOs". makefile() on an SSL connection pushes a buffering BIO on top of the SSL BIO and returns that, which then behaves like a Python file-like object. Even does readline()! See M2Crypto's BIO.py. 2.2b1 has a BIO member in PySSLObject but doesn't seem to use it. Nor do POW and pyOpenSSL, it appears. Well, that's only the SSL-related stuff that came to my mind readily. The Python development team has to decide how much of the above you want within Python, e.g., how much of the BIO and X.509 APIs to wrap. Not to mention PRNGs, secure memory handling and crypto-in-general. (Lastly, I'm pleased to note that M2Crypto is still ahead of the competition. Gotta keep moving, though! ;-) Cheers. -- Ng Pheng Siong <ngps@post1.com> * http://www.post1.com/home/ngps From barry@zope.com Mon Oct 29 17:53:23 2001 From: barry@zope.com (Barry A. Warsaw) Date: Mon, 29 Oct 2001 12:53:23 -0500 Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives References: <3BD9B5EC.119319A5@interet.com> <200110262034.QAA20802@cj20424-a.reston1.va.home.com> <3BDD6588.7CFEA7D6@interet.com> <200110291458.JAA13956@cj20424-a.reston1.va.home.com> <3BDD91CC.9F0ECB07@interet.com> <200110291739.MAA15773@cj20424-a.reston1.va.home.com> Message-ID: <15325.38803.700797.448117@anthem.wooz.org> >>>>> "GvR" == Guido van Rossum <guido@python.org> writes: GvR> Sounds good to me. Somebody please update the PEP. Jim, your last context diff was fine. Feel free to send me updates in the same format, and I'll apply them. -Barry From tim.one@home.com Mon Oct 29 18:14:56 2001 From: tim.one@home.com (Tim Peters) Date: Mon, 29 Oct 2001 13:14:56 -0500 Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives In-Reply-To: <3BDD91CC.9F0ECB07@interet.com> Message-ID: <LNBBLJKPBEHFEDALKOLCEEMKMDAA.tim.one@home.com> [James C. Ahlstrom] > The zip archive name is always inserted as the second item > in sys.path. The first always seems to be ''. Running p.py: import sys print `sys.path[0]` on Windows gives: C:\Code\python\PCbuild>python p.py '' C:\Code\python\PCbuild>cd .. C:\Code\python>pcbuild\python pcbuild\p.py 'pcbuild' C:\Code\python>pcbuild\python \code\python\pcbuild\p.py '\\code\\python\\pcbuild' C:\Code\python>cd pcbuild C:\Code\python\PCbuild>python .\p.py '.' C:\Code\python\PCbuild>move p.py ..\lib C:\CODE\PYTHON\PCBUILD\p.py => C:\CODE\PYTHON\lib\p.py [ok] C:\Code\python\PCbuild>python ..\lib\p.py '..\\lib' C:\Code\python\PCbuild> That is, the first entry is the path (relative or absolute!) to the directory in which the script being executed lives. From tim.one@home.com Mon Oct 29 18:22:10 2001 From: tim.one@home.com (Tim Peters) Date: Mon, 29 Oct 2001 13:22:10 -0500 Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives In-Reply-To: <200110291516.KAA14105@cj20424-a.reston1.va.home.com> Message-ID: <LNBBLJKPBEHFEDALKOLCKEMKMDAA.tim.one@home.com> [Guido] > I'd be happy to legislate uncompressed archives, except I'm worried that > not all tools commonly used to create zip archives make it easy to turn > off compression. FWIW, WinZip has an easy-to-get-at "compression: none" setting. I don't know whether that creates "an uncompress archive" from Jim's POV, but it sure doesn't save any space <wink>. From barry@zope.com Mon Oct 29 18:53:00 2001 From: barry@zope.com (Barry A. Warsaw) Date: Mon, 29 Oct 2001 13:53:00 -0500 Subject: [Python-Dev] [UPDATE] PEP 274, Dict Comprehensions Message-ID: <15325.42380.21091.468633@anthem.wooz.org> Attached is the latest version of PEP 274 for dictionary comprehensions, similar to list comprehensions. The first version of this PEP wasn't posted here, but several people saw the cvs checkin and sent in comments, which have now been incorporated. Enjoy, -Barry -------------------- snip snip -------------------- PEP: 274 Title: Dict Comprehensions Version: $Revision: 1.2 $ Last-Modified: $Date: 2001/10/29 18:46:59 $ Author: barry@zope.com (Barry A. Warsaw) Status: Draft Type: Standards Track Created: 25-Oct-2001 Python-Version: 2.3 Post-History: 29-Oct-2001 Abstract PEP 202 introduces a syntactical extension to Python called the "list comprehension"[1]. This PEP proposes a similar syntactical extension called the "dictionary comprehension" or "dict comprehension" for short. You can use dict comprehensions in ways very similar to list comprehensions, except that they produce Python dictionary objects instead of list objects. Proposed Solution Dict comprehensions are just like list comprehensions, except that you group the expression using curly braces instead of square braces. Also, the left part before the `for' keyword expresses both a key and a value, separated by a colon. (There is an optional part of this PEP that allows you to use a shortcut to express just the value.) The notation is specifically designed to remind you of list comprehensions as applied to dictionaries. Rationale There are times when you have some data arranged as a sequences of length-2 sequences, and you want to turn that into a dictionary. In Python 2.2, the dictionary() constructor will take an optional keyword argument that indicates specifically to interpret a sequences of length-2 sequences as key/value pairs, and turn them into a dictionary. However, the act of turning some data into a sequence of length-2 sequences can be inconvenient or inefficient from a memory or performance standpoint. Also, for some common operations, such as turning a list of things into a set of things for quick duplicate removal or set inclusion tests, a better syntax can help code clarity. As with list comprehensions, an explicit for loop can always be used (and in fact was the only way to do it in earlier versions of Python). But as with list comprehensions, dict comprehensions can provide a more syntactically succinct idiom that the traditional for loop. Examples >>> print {i : chr(65+i) for i in range(4)} {0 : 'A', 1 : 'B', 2 : 'C', 3 : 'D'} >>> print {k : v for k, v in someDict.iteritems()} == someDict.copy() 1 >>> print {x.lower() : 1 for x in list_of_email_addrs} {'barry@zope.com' : 1, 'barry@python.org' : 1, 'guido@python.org' : 1} >>> def invert(d): ... return {v : k for k, v in d.iteritems()} ... >>> d = {0 : 'A', 1 : 'B', 2 : 'C', 3 : 'D'} >>> print invert(d) {'A' : 0, 'B' : 1, 'C' : 2, 'D' : 3} Open Issues - There is one further shortcut we could adopt. Suppose we wanted to create a set of items, such as in the "list_of_email_addrs" example above. Here, we're simply taking the target of the for loop and turning that into the key for the dict comprehension. The assertion is that this would be a common idiom, so the shortcut below allows for an easy spelling of it, by allow us to omit the "key :" part of the left hand clause: >>> print {1 for x in list_of_email_addrs} {'barry@zope.com' : 1, 'barry@python.org' : 1, 'guido@python.org' : 1} Or say we wanted to map email addresses to the MX record handling their mail: >>> print {mx_for_addr(x) for x in list_of_email_addrs} {'barry@zope.com' : 'mail.zope.com', 'barry@python.org' : 'mail.python.org, 'guido@python.org' : 'mail.python.org, } Questions: what about nested loops? Where does the key come from? The shortcut probably doesn't save much typing, and comes at the expense of legibility, so it's of dubious value. - Should nested for loops be allowed? The following example, taken from an earlier revision of this PEP illustrates the problem: >>> print {k, v for k in range(4) for v in range(-4, 0, 1)} The intent of this example was to produce a mapping from a number to its negative, but this code doesn't work because -- as in list comprehensions -- the for loops are nested, not in parallel! So the value of this expression is actually {0: -1, 1: -1, 2: -1, 3: -1} which seems of dubious value. For symmetry with list comprehensions, perhaps this should be allowed, but it might be better to disallow this syntax. Implementation The semantics of dictionary comprehensions can actually be modeled in stock Python 2.2, by passing a list comprehension to the builtin dictionary constructor: >>> dictionary([(i, chr(65+i)) for i in range(4)]) This has two dictinct disadvantages from the proposed syntax though. First, it's isn't as legible as a dict comprehension. Second, it forces the programmer to create an in-core list object first, which could be expensive. References [1] PEP 202, List Comprehensions http://www.python.org/peps/pep-0202.html Copyright This document has been placed in the public domain. Local Variables: mode: indented-text indent-tabs-mode: nil fill-column: 70 End: From guido@python.org Mon Oct 29 19:08:17 2001 From: guido@python.org (Guido van Rossum) Date: Mon, 29 Oct 2001 14:08:17 -0500 Subject: [Python-Dev] [UPDATE] PEP 274, Dict Comprehensions In-Reply-To: Your message of "Mon, 29 Oct 2001 13:53:00 EST." <15325.42380.21091.468633@anthem.wooz.org> References: <15325.42380.21091.468633@anthem.wooz.org> Message-ID: <200110291908.OAA16263@cj20424-a.reston1.va.home.com> > - There is one further shortcut we could adopt. Suppose we wanted > to create a set of items, such as in the "list_of_email_addrs" > example above. Here, we're simply taking the target of the for > loop and turning that into the key for the dict comprehension. Hm, I don't like this. I think it's confusing: you really have to think about whether {x for x in <whatever>} produces a dictionary whose keys are in <whatever> or one whose values are in <whatever>. > - Should nested for loops be allowed? The following example, > taken from an earlier revision of this PEP illustrates the > problem: > > >>> print {k, v for k in range(4) for v in range(-4, 0, 1)} Don't you mean {k: v for ...}? The second range() also seems like you meant range(0, -4, -1). > The intent of this example was to produce a mapping from a > number to its negative, but this code doesn't work because -- as > in list comprehensions -- the for loops are nested, not in > parallel! So the value of this expression is actually > > {0: -1, 1: -1, 2: -1, 3: -1} > > which seems of dubious value. For symmetry with list > comprehensions, perhaps this should be allowed, but it might be > better to disallow this syntax. Nested for loops can be useful when used properly: {(k, v): k+v for k in range(4) for v in range(4)} Your example should have been expressed using: {k: v for k, v in zip(range(4), range(0, -4, -1))} --Guido van Rossum (home page: http://www.python.org/~guido/) From bckfnn@worldonline.dk Mon Oct 29 19:15:00 2001 From: bckfnn@worldonline.dk (Finn Bock) Date: Mon, 29 Oct 2001 19:15:00 GMT Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives In-Reply-To: <200110291513.KAA14082@cj20424-a.reston1.va.home.com> References: <3BD9B5EC.119319A5@interet.com> <3bd9d419.10386565@mail.wanadoo.dk> <3BDD691B.2599AAF4@interet.com> <200110291513.KAA14082@cj20424-a.reston1.va.home.com> Message-ID: <3bddaa12.11250086@mail.wanadoo.dk> [JimA] > I am confused. Zip archives are equivalant to subdirectories, so there > is no requirement to have anything at the top level. Your example seems > to imply a second search path besides sys.path. [GvR] >I think Finn simply means that the zipfile may have some redundant >initial suffix to all filenames (e.g. "Lib/") ... Yes, that was it! regards, finn From bckfnn@worldonline.dk Mon Oct 29 19:16:21 2001 From: bckfnn@worldonline.dk (Finn Bock) Date: Mon, 29 Oct 2001 19:16:21 GMT Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives In-Reply-To: <3BDD6B74.D4E19953@interet.com> References: <3BD9B5EC.119319A5@interet.com> <3bd9d419.10386565@mail.wanadoo.dk> <200110270223.WAA23737@cj20424-a.reston1.va.home.com> <3bdbfda4.3697676@mail.wanadoo.dk> <3BDD6B74.D4E19953@interet.com> Message-ID: <3bdda1df.9151318@mail.wanadoo.dk> [me] > myself), I just didn't dare to put such a limit on my jython > implementation. [James C. Ahlstrom] >I don't understand the jpython implementation, so please point out >all problems so we can fix them now. A naive implementation (like the first one I made for Jython) does not try to handle the cleanup issues. For example, how much memory have been consumed and not freed after the last statement in this little silly program: import sys, zipfile, time zip = zipfile.ZipFile("archive.zip", "w") for i in range(10000): entry = zipfile.ZipInfo() entry.filename = "module%06d.py" % i entry.date_time = time.gmtime(time.time()) zip.writestr(entry, "# Not used\n") zip.close() sys.path.append("archive.zip") try: import notfound except ImportError: pass sys.path.pop() If I read your preliminary patch correctly, the 10000 entries will remain in the ArchiveMembers dict forever and that is perfectly fine after Guido comments on sys.path being mostly a static feature. Jython manage to clean the member-cache when an archive is removed from sys.path but is was quite tricky to achieve. We do it by replacing zip entries in sys.path (like "archive.zip") with a subclass of string (SyspathArchive) that also holds a reference to the member cache. From pythons POV the sys.path entry is still a string with the same value (but with a different id). regards, finn From tim.one@home.com Mon Oct 29 19:14:39 2001 From: tim.one@home.com (Tim Peters) Date: Mon, 29 Oct 2001 14:14:39 -0500 Subject: [Python-Dev] [UPDATE] PEP 274, Dict Comprehensions In-Reply-To: <15325.42380.21091.468633@anthem.wooz.org> Message-ID: <LNBBLJKPBEHFEDALKOLCOENAMDAA.tim.one@home.com> [Barry] > Rationale > > There are times when you have some data arranged as a sequences of > length-2 sequences, and you want to turn that into a dictionary. > In Python 2.2, the dictionary() constructor will take an optional > keyword argument that indicates specifically to interpret a > sequences of length-2 sequences as key/value pairs, and turn them > into a dictionary. This is implemented now, but in a different way. Suggested rewording: In Python 2.2, the dictionary() constructor accepts an argument that is a sequence of length-2 sequences, used as (key, value) pairs to initialize a new dictionary object. BTW, and not meaning to hijack your PEP <wink>, should dict.update() accept such an argument too? I didn't add it because d.update(dictionary(such_an_argument)) seemed "almost good enough". BTW2, are we going to rename "dictionary" to "dict" before 2.2b2? Before 2.2, "dict" was universally used on c.l.py to mean dictionary, and I'm at least +0 on adopting that for official 2.2 use. From barry@zope.com Mon Oct 29 19:22:58 2001 From: barry@zope.com (Barry A. Warsaw) Date: Mon, 29 Oct 2001 14:22:58 -0500 Subject: [Python-Dev] [UPDATE] PEP 274, Dict Comprehensions References: <15325.42380.21091.468633@anthem.wooz.org> <LNBBLJKPBEHFEDALKOLCOENAMDAA.tim.one@home.com> Message-ID: <15325.44178.193256.554626@anthem.wooz.org> >>>>> "TP" == Tim Peters <tim.one@home.com> writes: TP> BTW, and not meaning to hijack your PEP <wink>, should TP> dict.update() accept such an argument too? I didn't add it TP> because TP> d.update(dictionary(such_an_argument)) TP> seemed "almost good enough". Agreed. But either way, I still there there is utility in a dict comprehension. TP> BTW2, are we going to rename "dictionary" to "dict" before TP> 2.2b2? Before 2.2, "dict" was universally used on c.l.py to TP> mean dictionary, and I'm at least +0 on adopting that for TP> official 2.2 use. I wouldn't keep them both though. Use one or the other. -Barry From guido@python.org Mon Oct 29 19:27:32 2001 From: guido@python.org (Guido van Rossum) Date: Mon, 29 Oct 2001 14:27:32 -0500 Subject: [Python-Dev] Dict API refinements (was PEP 274) In-Reply-To: Your message of "Mon, 29 Oct 2001 14:14:39 EST." <LNBBLJKPBEHFEDALKOLCOENAMDAA.tim.one@home.com> References: <LNBBLJKPBEHFEDALKOLCOENAMDAA.tim.one@home.com> Message-ID: <200110291927.OAA16469@cj20424-a.reston1.va.home.com> > BTW, and not meaning to hijack your PEP <wink>, should dict.update() accept > such an argument too? I didn't add it because > > d.update(dictionary(such_an_argument)) > > seemed "almost good enough". Agreed; no need to add this to update(). > BTW2, are we going to rename "dictionary" to "dict" before 2.2b2? > Before 2.2, "dict" was universally used on c.l.py to mean > dictionary, and I'm at least +0 on adopting that for official 2.2 > use. Sounds good to me. Should we adopt "dict" as an alias and keep "dictionary" as the official name, or vice versa, or simply eradicate dictionary and introduce dict in 2.2b2? It's up to you to implement this, I have some other things I need to get done first. --Guido van Rossum (home page: http://www.python.org/~guido/) From tim.one@home.com Mon Oct 29 19:45:47 2001 From: tim.one@home.com (Tim Peters) Date: Mon, 29 Oct 2001 14:45:47 -0500 Subject: [Python-Dev] test_asynchat.py broken on Windows Message-ID: <LNBBLJKPBEHFEDALKOLCGENGMDAA.tim.one@home.com> Here on Windows, current CVS: C:\Code\python\PCbuild>python ../lib/test/test_asynchat.py Traceback (most recent call last): File "../lib/test/test_asynchat.py", line 58, in ? main() File "../lib/test/test_asynchat.py", line 53, in main c = echo_client() File "../lib/test/test_asynchat.py", line 34, in __init__ self.connect((HOST, PORT)) File "C:\CODE\PYTHON\lib\asyncore.py", line 311, in connect raise socket.error, err socket.error: 2 The DOS box is frozen at this point, but not accumulating any cycles. Worked 12 hours ago. What about Linux? From thomas.heller@ion-tof.com Mon Oct 29 19:54:31 2001 From: thomas.heller@ion-tof.com (Thomas Heller) Date: Mon, 29 Oct 2001 20:54:31 +0100 Subject: [Python-Dev] class dict of subtyped types Message-ID: <046c01c160b3$875c7340$e000a8c0@thomasnotebook> C:\>python Python 2.2b1+ (#25, Oct 19 2001, 14:30:05) [MSC 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> class INT(int): pass ... >>> INT.__dict__ <dict-proxy object at 0x00769680> >>> INT.__dict__.update Traceback (most recent call last): File "<stdin>", line 1, in ? AttributeError: 'dict-proxy' object has no attribute 'update' >>> dir(INT.__dict__) ['__class__', '__contains__', '__delattr__', '__getattribute__', '__getitem__', '__hash__' , '__init__', '__iter__', '__len__', '__new__', '__reduce__', '__repr__', '__setattr__', ' __str__', 'copy', 'get', 'has_key', 'items', 'keys', 'values'] >>> Other dict attributes are missing as well in the dict-proxy. Is this intentional? Thomas From tim.one@home.com Mon Oct 29 19:55:36 2001 From: tim.one@home.com (Tim Peters) Date: Mon, 29 Oct 2001 14:55:36 -0500 Subject: [Python-Dev] RE: Dict API refinements (was PEP 274) In-Reply-To: <200110291927.OAA16469@cj20424-a.reston1.va.home.com> Message-ID: <LNBBLJKPBEHFEDALKOLCOENHMDAA.tim.one@home.com> [Tim] >> BTW2, are we going to rename "dictionary" to "dict" before 2.2b2? >> Before 2.2, "dict" was universally used on c.l.py to mean >> dictionary, and I'm at least +0 on adopting that for official 2.2 >> use. [Guido] > Sounds good to me. Should we adopt "dict" as an alias and keep > "dictionary" as the official name, or vice versa, or simply eradicate > dictionary and introduce dict in 2.2b2? s/dictionary/dict/g is my preference: TOOWTDI. > It's up to you to implement this, I have some other things I need to > get done first. That's fine -- I can't think of anything needed I don't know how to do with ease (heck, I even know where the dictionary() docstring lives <wink>), except for changing the descr tutorial accordingly on python.org. From fdrake@acm.org Mon Oct 29 19:53:49 2001 From: fdrake@acm.org (Fred L. Drake, Jr.) Date: Mon, 29 Oct 2001 14:53:49 -0500 Subject: [Python-Dev] RE: Dict API refinements (was PEP 274) In-Reply-To: <LNBBLJKPBEHFEDALKOLCOENHMDAA.tim.one@home.com> References: <200110291927.OAA16469@cj20424-a.reston1.va.home.com> <LNBBLJKPBEHFEDALKOLCOENHMDAA.tim.one@home.com> Message-ID: <15325.46029.119091.651467@grendel.zope.com> Tim Peters writes: > That's fine -- I can't think of anything needed I don't know how to do with > ease (heck, I even know where the dictionary() docstring lives <wink>), > except for changing the descr tutorial accordingly on python.org. I'll volunteer to take care of that one. -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> PythonLabs at Zope Corporation From guido@python.org Mon Oct 29 19:59:51 2001 From: guido@python.org (Guido van Rossum) Date: Mon, 29 Oct 2001 14:59:51 -0500 Subject: [Python-Dev] RE: Dict API refinements (was PEP 274) In-Reply-To: Your message of "Mon, 29 Oct 2001 14:55:36 EST." <LNBBLJKPBEHFEDALKOLCOENHMDAA.tim.one@home.com> References: <LNBBLJKPBEHFEDALKOLCOENHMDAA.tim.one@home.com> Message-ID: <200110291959.OAA16703@cj20424-a.reston1.va.home.com> > [Tim] > >> BTW2, are we going to rename "dictionary" to "dict" before 2.2b2? > >> Before 2.2, "dict" was universally used on c.l.py to mean > >> dictionary, and I'm at least +0 on adopting that for official 2.2 > >> use. > > [Guido] > > Sounds good to me. Should we adopt "dict" as an alias and keep > > "dictionary" as the official name, or vice versa, or simply eradicate > > dictionary and introduce dict in 2.2b2? [Tim] > s/dictionary/dict/g is my preference: TOOWTDI. [Guido] OK, let's do that. > > It's up to you to implement this, I have some other things I need to > > get done first. > > That's fine -- I can't think of anything needed I don't know how to do with > ease (heck, I even know where the dictionary() docstring lives <wink>), > except for changing the descr tutorial accordingly on python.org. OK, I'll fix that one. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Mon Oct 29 20:04:12 2001 From: guido@python.org (Guido van Rossum) Date: Mon, 29 Oct 2001 15:04:12 -0500 Subject: [Python-Dev] class dict of subtyped types In-Reply-To: Your message of "Mon, 29 Oct 2001 20:54:31 +0100." <046c01c160b3$875c7340$e000a8c0@thomasnotebook> References: <046c01c160b3$875c7340$e000a8c0@thomasnotebook> Message-ID: <200110292004.PAA16731@cj20424-a.reston1.va.home.com> > C:\>python > Python 2.2b1+ (#25, Oct 19 2001, 14:30:05) [MSC 32 bit (Intel)] on win32 > Type "help", "copyright", "credits" or "license" for more information. > >>> class INT(int): pass > ... > >>> INT.__dict__ > <dict-proxy object at 0x00769680> > >>> INT.__dict__.update > Traceback (most recent call last): > File "<stdin>", line 1, in ? > AttributeError: 'dict-proxy' object has no attribute 'update' > >>> dir(INT.__dict__) > ['__class__', '__contains__', '__delattr__', '__getattribute__', '__getitem__', '__hash__' > , '__init__', '__iter__', '__len__', '__new__', '__reduce__', '__repr__', '__setattr__', ' > __str__', 'copy', 'get', 'has_key', 'items', 'keys', 'values'] > >>> > > Other dict attributes are missing as well in the dict-proxy. > Is this intentional? The dict-proxy type is intended to provide a read-only proxy, so the dict-proxy is consciously lacking the update, clear, popitem and setdefault methods. It also seems to be missing the comparison operations; that's an oversight. --Guido van Rossum (home page: http://www.python.org/~guido/) From thomas.heller@ion-tof.com Mon Oct 29 20:09:48 2001 From: thomas.heller@ion-tof.com (Thomas Heller) Date: Mon, 29 Oct 2001 21:09:48 +0100 Subject: [Python-Dev] class dict of subtyped types References: <046c01c160b3$875c7340$e000a8c0@thomasnotebook> <200110292004.PAA16731@cj20424-a.reston1.va.home.com> Message-ID: <055601c160b5$a9473dd0$e000a8c0@thomasnotebook> From: "Guido van Rossum" <guido@python.org> > > C:\>python > > Python 2.2b1+ (#25, Oct 19 2001, 14:30:05) [MSC 32 bit (Intel)] on win32 > > Type "help", "copyright", "credits" or "license" for more information. > > >>> class INT(int): pass > > ... > > >>> INT.__dict__ > > <dict-proxy object at 0x00769680> > > >>> INT.__dict__.update > > Traceback (most recent call last): > > File "<stdin>", line 1, in ? > > AttributeError: 'dict-proxy' object has no attribute 'update' > > >>> dir(INT.__dict__) > > ['__class__', '__contains__', '__delattr__', '__getattribute__', '__getitem__', '__hash__' > > , '__init__', '__iter__', '__len__', '__new__', '__reduce__', '__repr__', '__setattr__', ' > > __str__', 'copy', 'get', 'has_key', 'items', 'keys', 'values'] > > >>> > > > > Other dict attributes are missing as well in the dict-proxy. > > Is this intentional? > > The dict-proxy type is intended to provide a read-only proxy, so the > dict-proxy is consciously lacking the update, clear, popitem and > setdefault methods. It also seems to be missing the comparison > operations; that's an oversight. > But adding attributes to the INT class above after creation is allowed, I hope? At least it currently works. Thomas From guido@python.org Mon Oct 29 20:12:19 2001 From: guido@python.org (Guido van Rossum) Date: Mon, 29 Oct 2001 15:12:19 -0500 Subject: [Python-Dev] class dict of subtyped types In-Reply-To: Your message of "Mon, 29 Oct 2001 21:09:48 +0100." <055601c160b5$a9473dd0$e000a8c0@thomasnotebook> References: <046c01c160b3$875c7340$e000a8c0@thomasnotebook> <200110292004.PAA16731@cj20424-a.reston1.va.home.com> <055601c160b5$a9473dd0$e000a8c0@thomasnotebook> Message-ID: <200110292012.PAA16819@cj20424-a.reston1.va.home.com> > > The dict-proxy type is intended to provide a read-only proxy, so the > > dict-proxy is consciously lacking the update, clear, popitem and > > setdefault methods. It also seems to be missing the comparison > > operations; that's an oversight. > > > But adding attributes to the INT class above after creation > is allowed, I hope? At least it currently works. Yes, but you have to use the attribute notation: INT.foo = ... this is trapped so that certain magic happens automatically when you set attributes that override operators, e.g. INT.__add__ = lambda ... --Guido van Rossum (home page: http://www.python.org/~guido/) From jim@interet.com Mon Oct 29 20:57:38 2001 From: jim@interet.com (James C. Ahlstrom) Date: Mon, 29 Oct 2001 15:57:38 -0500 Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives References: <LNBBLJKPBEHFEDALKOLCEEMKMDAA.tim.one@home.com> Message-ID: <3BDDC2C2.E26289D5@interet.com> Tim Peters wrote: > > That is, the first entry is the path (relative or absolute!) to the > directory in which the script being executed lives. Right you are! JimA From jeremy@alum.mit.edu Mon Oct 29 23:20:14 2001 From: jeremy@alum.mit.edu (Jeremy Hylton) Date: Mon, 29 Oct 2001 18:20:14 -0500 (EST) Subject: [Python-Dev] Re: test_asynchat.py broken on Windows Message-ID: <200110292320.SAA11414@newman.concentric.net> I think there are at least three things wrong, depending on what you count as wrong. The test is hanging because it spawns a thread, which Python will prevent Python from exiting until the thread exits. The thread doesn't exit because it's waiting for input from the client. The client doesn't send it any input because of the exception. thing wrong #1: It would probably make sense to change test_asynchat.py to use a daemon thread so that it can't prevent the test suite from exiting. thing wrong #2: I don't think the ENONET being returned by connect_ex() makes any sense. My guess is that connect_ex() is (and always has been) broken on Windows. Could you apply this patch and see what happens: Index: socketmodule.c =================================================================== RCS file: /cvsroot/python/python/dist/src/Modules/socketmodule.c,v retrieving revision 1.190 diff -c -c -r1.190 socketmodule.c *** socketmodule.c 2001/10/28 12:31:33 1.190 --- socketmodule.c 2001/10/29 23:14:18 *************** *** 1268,1274 **** --- 1268,1278 ---- res = connect(s->sock_fd, addr, addrlen); Py_END_ALLOW_THREADS if (res != 0) + #ifdef MS_WINDOWS + res = WSAGetLastError(); + #else res = errno; + #endif return PyInt_FromLong((long) res); } thing wrong #3: From the winsock documentation for connect(), it looks like there are more errors we should be catching in the test that includes EWOULDBLOCK and EALREADY. I don't think this would affect your test, though. thing wrong #4: It doesn't look to me like the connect() call in test_asynchat.py is guaranteed to succeed. It should succeed almost every time, but there's some small chance it will fail. Perhaps the test should be more robust about that. That's four things, but I don't think the last thing is all that wrong. Jeremy From Anthony Baxter <anthony@interlink.com.au> Mon Oct 29 23:53:41 2001 From: Anthony Baxter <anthony@interlink.com.au> (Anthony Baxter) Date: Tue, 30 Oct 2001 10:53:41 +1100 Subject: [Python-Dev] Fixing send() In-Reply-To: Message from Anthony Baxter <anthony@interlink.com.au> of "Thu, 25 Oct 2001 16:24:20 +1000." Message-ID: <200110292353.f9TNrfo27736@mbuna.arbhome.com.au> > Will do - I plan to check in a bunch of fixes tomorrow morning. (continuing > gdb hell today). Just an update to this - my plans were wrecked by a cablemodem outage, however I'm spending tomorrow on branch mgmt - work is keen for me to take some of the 6 weeks leave I've built up, so I'm taking a few days off to work on this. As far as trying to delegate checkins of code - would people prefer I post a big list here, mail the original authors of the patches, or just do it myself? Anthony From greg@cosc.canterbury.ac.nz Tue Oct 30 00:39:41 2001 From: greg@cosc.canterbury.ac.nz (Greg Ewing) Date: Tue, 30 Oct 2001 13:39:41 +1300 (NZDT) Subject: [Python-Dev] Dict API refinements (was PEP 274) In-Reply-To: <200110291927.OAA16469@cj20424-a.reston1.va.home.com> Message-ID: <200110300039.NAA09256@s454.cosc.canterbury.ac.nz> Guido van Rossum <guido@python.org>: > Should we adopt "dict" as an alias and keep > "dictionary" as the official name, or vice versa, or simply eradicate > dictionary and introduce dict in 2.2b2? I'd say eradicate "dictionary". There Should Only Be One Way To Spell It. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From tim.one@home.com Tue Oct 30 01:37:49 2001 From: tim.one@home.com (Tim Peters) Date: Mon, 29 Oct 2001 20:37:49 -0500 Subject: [Python-Dev] RE: test_asynchat.py broken on Windows In-Reply-To: <200110292320.SAA11414@newman.concentric.net> Message-ID: <LNBBLJKPBEHFEDALKOLCGEPEMDAA.tim.one@home.com> [Jeremy Hylton] > I think there are at least three things wrong, depending on what you > count as wrong. I would have guessed five, if you were counting Windows too <wink>. > The test is hanging because it spawns a thread, which Python will > prevent Python from exiting until the thread exits. The thread > doesn't exit because it's waiting for input from the client. The > client doesn't send it any input because of the exception. > > thing wrong #1: It would probably make sense to change > test_asynchat.py to use a daemon thread so that it can't prevent the > test suite from exiting. I don't much care about this. > thing wrong #2: I don't think the ENONET being returned by > connect_ex() makes any sense. My guess is that connect_ex() is (and > always has been) broken on Windows. Could you apply this patch and > see what happens: > ... Fixed the problem, so I checked it in. Thanks! > thing wrong #3: From the winsock documentation for connect(), it looks > like there are more errors we should be catching in the test that > includes EWOULDBLOCK and EALREADY. I don't think this would affect > your test, though. I'm not sure I parsed that as intended; asyncore.dispatcher.connect (in the traceback when test_asynchat failed) is already checking for EINPROGRESS, EALREADY, EWOULDBLOCK, 0 and EISCONN returns from connect_ex. But, for posterity, I will inscribe my socket programming knowledge in the period at the end of this sentence. > thing wrong #4: It doesn't look to me like the connect() call in > test_asynchat.py is guaranteed to succeed. It should succeed almost > every time, but there's some small chance it will fail. Perhaps the > test should be more robust about that. FWIW, I've never seen it fail before. > That's four things, but I don't think the last thing is all that wrong. Here's a fifth: def main(): s = echo_server() s.start() time.sleep(1) # Give server time to initialize c = echo_client() ... time.sleep()-is-not-a-synchronization-gimmick-ly y'rs - tim From tim.one@home.com Tue Oct 30 02:01:40 2001 From: tim.one@home.com (Tim Peters) Date: Mon, 29 Oct 2001 21:01:40 -0500 Subject: [Python-Dev] Fixing send() In-Reply-To: <200110292353.f9TNrfo27736@mbuna.arbhome.com.au> Message-ID: <LNBBLJKPBEHFEDALKOLCCEPFMDAA.tim.one@home.com> [Anthony Baxter] > Just an update to this - my plans were wrecked by a cablemodem outage, > however I'm spending tomorrow on branch mgmt - work is keen for me to > take some of the 6 weeks leave I've built up, so I'm taking a few days > off to work on this. We have similar views on what constitutes "a vacation" <wink>. > As far as trying to delegate checkins of code - would people prefer I > post a big list here, mail the original authors of the patches, or just > do it myself? Do it yourself so far as is humanly possible. That way (a) it will get done; and (b) the developers won't come to resent bugfix releases (if we had time to spare for this, we wouldn't ask for volunteers). it's-a-job-as-well-as-an-easy-way-to-burn-excess-vacation-days-ly y'rs - tim From guido@python.org Tue Oct 30 02:03:33 2001 From: guido@python.org (Guido van Rossum) Date: Mon, 29 Oct 2001 21:03:33 -0500 Subject: [Python-Dev] Fixing send() In-Reply-To: Your message of "Tue, 30 Oct 2001 10:53:41 +1100." <200110292353.f9TNrfo27736@mbuna.arbhome.com.au> References: <200110292353.f9TNrfo27736@mbuna.arbhome.com.au> Message-ID: <200110300203.VAA19633@cj20424-a.reston1.va.home.com> > As far as trying to delegate checkins of code - would people prefer I > post a big list here, mail the original authors of the patches, or just > do it myself? In general, trying to reach the original authors will just add delays. Do it yourself, or if you're not sure, post here. --Guido van Rossum (home page: http://www.python.org/~guido/) From greg@cosc.canterbury.ac.nz Tue Oct 30 02:25:09 2001 From: greg@cosc.canterbury.ac.nz (Greg Ewing) Date: Tue, 30 Oct 2001 15:25:09 +1300 (NZDT) Subject: [Python-Dev] [UPDATE] PEP 274, Dict Comprehensions In-Reply-To: <15325.42380.21091.468633@anthem.wooz.org> Message-ID: <200110300225.PAA09272@s454.cosc.canterbury.ac.nz> +1 generally on dict comprehensions, but: > >>> print {1 for x in list_of_email_addrs} > {'barry@zope.com' : 1, 'barry@python.org' : 1, 'guido@python.org' : 1} -1 on this bit. It's not at all clear what it should mean, and the saving over writing it out explicitly, i.e. {x:1 for x in list_of_email_addrs} is vanishingly small. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From guido@python.org Tue Oct 30 02:28:47 2001 From: guido@python.org (Guido van Rossum) Date: Mon, 29 Oct 2001 21:28:47 -0500 Subject: [Python-Dev] [UPDATE] PEP 274, Dict Comprehensions In-Reply-To: Your message of "Tue, 30 Oct 2001 15:25:09 +1300." <200110300225.PAA09272@s454.cosc.canterbury.ac.nz> References: <200110300225.PAA09272@s454.cosc.canterbury.ac.nz> Message-ID: <200110300228.VAA21968@cj20424-a.reston1.va.home.com> > +1 generally on dict comprehensions, but: > > > >>> print {1 for x in list_of_email_addrs} > > {'barry@zope.com' : 1, 'barry@python.org' : 1, 'guido@python.org' : 1} > > -1 on this bit. It's not at all clear what it should mean, > and the saving over writing it out explicitly, i.e. > > {x:1 for x in list_of_email_addrs} > > is vanishingly small. You're getting better at channeling me. :-) --Guido van Rossum (home page: http://www.python.org/~guido/) From fdrake@acm.org Tue Oct 30 06:23:32 2001 From: fdrake@acm.org (Fred L. Drake) Date: Tue, 30 Oct 2001 01:23:32 -0500 (EST) Subject: [Python-Dev] [development doc updates] Message-ID: <20011030062332.77F4928697@cj42289-a.reston1.va.home.com> The development version of the documentation has been updated: http://python.sourceforge.net/devel-docs/ Re-arranged the material in Chapter 2 (built-in things) to make it easier to start off with. Functions come before Types and Exceptions, and file objects are promoted one level in the outline, making them easier to find (they now appear in the table of contents instead of being hidden in the "Other Objects" category). From mal@lemburg.com Tue Oct 30 08:25:28 2001 From: mal@lemburg.com (M.-A. Lemburg) Date: Tue, 30 Oct 2001 09:25:28 +0100 Subject: [Python-Dev] Slices and "==" optimization Message-ID: <3BDE63F8.541E04B2@lemburg.com> While writing yet another XML parser I have come across two things which would seem nice to have: 1. Even though you can construct slice objects in Python using slice(left,right), you can't really do anything with them; at least not on the standard types (Numeric's arrays work with them just fine). Couldn't we add some functionality which makes them compatible to lists and tuples (and perhaps even in a generic way for other sequences using PySequence_GetItem()) too ? I thinking of extending the __getitem__ hook to accept slice objects. You could then maintain slices to an object in say a list and apply them to the underlying text as needed, e.g. s = slice(1,4) l = range(10) l[s] == [1, 2, 3] 2. Looking through the code for '==' the optimization for 'a == a' seems too far down the call stack. Since interning strings makes this case rather common, I'd suggest to lift the optimization into the ceval loop (right along side with the INT op INT optimization). Thoughts ? -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Consulting & Company: http://www.egenix.com/ Python Software: http://www.lemburg.com/python/ From martin@v.loewis.de Tue Oct 30 08:42:12 2001 From: martin@v.loewis.de (Martin v. Loewis) Date: Tue, 30 Oct 2001 09:42:12 +0100 Subject: [Python-Dev] Slices and "==" optimization Message-ID: <200110300842.f9U8gCX01628@mira.informatik.hu-berlin.de> > I thinking of extending the __getitem__ hook to accept slice objects Please have a look at http://sourceforge.net/tracker/?func=detail&aid=459235&group_id=5470&atid=105470 It's a known problem, but Guido won't fix it anytime soon. So I guess contributions are welcome. > Looking through the code for '==' the optimization for 'a == a' > seems too far down the call stack. Since interning strings makes > this case rather common, I'd suggest to lift the optimization into > the ceval loop As with all optimizations: Patches to just improve the performance are unacceptable unless accompanied by some hard numbers to show that they really do have the desired effect. History shows that patches to just improve the performance may be unacceptable even if accompanied by such numbers. Normally, Tim will construct a scenario where the patch causes a slow-down, and argue that this is a common scenario :-) Regards, Martin From fredrik@pythonware.com Tue Oct 30 08:56:07 2001 From: fredrik@pythonware.com (Fredrik Lundh) Date: Tue, 30 Oct 2001 09:56:07 +0100 Subject: [Python-Dev] RE: test_asynchat.py broken on Windows References: <LNBBLJKPBEHFEDALKOLCGEPEMDAA.tim.one@home.com> Message-ID: <00c901c16120$b954e180$ced241d5@hagrid> > [Jeremy Hylton] > > I think there are at least three things wrong, depending on what you > > count as wrong. not sure jeremy gets mail sent to his sourceforge address, but maybe someone else can explain how his recent patch preserves the original behaviour. after the patch, connect will always raise an exception unless the error code is one of the three listed code. before the patch, it will not raise an exception if the connection succeeds during the connect itself (i.e. if people use asyncore with blocking sockets, or if they're using a hypothetical transport that can do an immediate connect). how about adding an "else:" before that last raise? def connect (self, address): self.connected = 0 ! # XXX why not use connect_ex? ! try: ! self.socket.connect (address) ! except socket.error, why: ! if why[0] in (EINPROGRESS, EALREADY, EWOULDBLOCK): ! return ! else: ! raise socket.error, why ! self.addr = address ! self.connected = 1 ! self.handle_connect() def accept (self): --- 302,313 ---- def connect (self, address): self.connected = 0 ! err = self.socket.connect_ex(address) ! if err in (EINPROGRESS, EALREADY, EWOULDBLOCK): ! return ! if err in (0, EISCONN): ! self.addr = address ! self.connected = 1 ! self.handle_connect() ! raise socket.error, err def accept (self): From mal@lemburg.com Tue Oct 30 08:55:09 2001 From: mal@lemburg.com (M.-A. Lemburg) Date: Tue, 30 Oct 2001 09:55:09 +0100 Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives References: <3BD9B5EC.119319A5@interet.com> <3BDB1EB0.C2A9B1FF@lemburg.com> <3BDD6D66.675A4F3C@interet.com> Message-ID: <3BDE6AED.25D67499@lemburg.com> "James C. Ahlstrom" wrote: > > "M.-A. Lemburg" wrote: > > > 1. Why are .py files being outlawed ? > > My reason was that if *.py[co] are missing or out of date, > zip importing will be slow and users won't figure > out what is wrong. Generally I favor user-proof features > over expert features. I prefer things which either "Just Work" > or fail spectacularly. But I am open to changing this. If you don't include the *.py file in the archive, chances are high that tracebacks will no longer print out as they do today (another problem here is that the filename being used in the code object will probably not be found... not sure whether we can fix this one). By only looking at the .pyc or .pyo you'll also introduce a Python version problem into the ZIP-archive: the magic in these files changs rather frequently and e.g. a more recent release of Python won't be able to load these from a ZIP-archive which only contains .pycs from a compile by a less recent version. > > 2. Where's the C implementation you mention in the PEP ? > > Software is like pancakes, you should always > throw the first one away. I will post it if you want, > but it is not done. That's true :-) Still, I'd like a chance to at least look at the impact this has on import.c and of course play with it so that I can test it in everyday situations. > > 3. Would it be possible to ship zlib together with Python ? > > (the zlib license should allow this and I don't think that > > the code size is too big) > > OK by me. But uncompressed zip archives work, and may even > be faster than conpressed archives. Even if uncompressed archive would work faster, the compiled libz is only 70k on Linux and I think we could solve a great number of (zlib version) problems by including zlib in Python. It's one of those basic tools you need frequently, but is even more frequently not configured as Python module :-( Since it is already included in the Windows builds, I guess adding it to core for Unix and Macs too wouldn't hurt all that much. It would also save you the trouble of maintaining the code for scanning uncompressed zip-archives in your ZIP import code, so we win on two counts :-) -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Consulting & Company: http://www.egenix.com/ Python Software: http://www.lemburg.com/python/ From mal@lemburg.com Tue Oct 30 10:12:43 2001 From: mal@lemburg.com (M.-A. Lemburg) Date: Tue, 30 Oct 2001 11:12:43 +0100 Subject: [Python-Dev] Slices and "==" optimization References: <200110300842.f9U8gCX01628@mira.informatik.hu-berlin.de> Message-ID: <3BDE7D1B.4850329C@lemburg.com> "Martin v. Loewis" wrote: > > > I thinking of extending the __getitem__ hook to accept slice objects > > Please have a look at > > http://sourceforge.net/tracker/?func=detail&aid=459235&group_id=5470&atid=105470 > > It's a known problem, but Guido won't fix it anytime soon. So I guess > contributions are welcome. Good to know, thanks. > > Looking through the code for '==' the optimization for 'a == a' > > seems too far down the call stack. Since interning strings makes > > this case rather common, I'd suggest to lift the optimization into > > the ceval loop > > As with all optimizations: Patches to just improve the performance are > unacceptable unless accompanied by some hard numbers to show that they > really do have the desired effect. > > History shows that patches to just improve the performance may be > unacceptable even if accompanied by such numbers. Normally, Tim will > construct a scenario where the patch causes a slow-down, and argue > that this is a common scenario :-) Sounds like you have experience ;-) As with all micro-optimizations, the win usually only shows up in some applications. The one I'm writing uses lot's of if tagtype == 'starttag': if tagtype == 'endtag': ... Since the 'starttag' and 'endtag' strings are interned and I'm also interning the strings which are stored in tagtype, I'd like to benefit from the fact that the compare will actually work as 'is'-compare. However, I don't want to use 'is' because I consider interning only an optimization and not a feature of the language. That's why I would like the simple if (v == w) return 0; integrated into the ceval loop right along the INT-compare optimization. This may also help in some other situations where objects are shared. -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Consulting & Company: http://www.egenix.com/ Python Software: http://www.lemburg.com/python/ From guido@python.org Tue Oct 30 13:50:47 2001 From: guido@python.org (Guido van Rossum) Date: Tue, 30 Oct 2001 08:50:47 -0500 Subject: [Python-Dev] Slices and "==" optimization In-Reply-To: Your message of "Tue, 30 Oct 2001 09:25:28 +0100." <3BDE63F8.541E04B2@lemburg.com> References: <3BDE63F8.541E04B2@lemburg.com> Message-ID: <200110301350.IAA24747@cj20424-a.reston1.va.home.com> > 1. Even though you can construct slice objects in Python > using slice(left,right), you can't really do anything > with them; at least not on the standard types (Numeric's > arrays work with them just fine). > > Couldn't we add some functionality which makes them > compatible to lists and tuples (and perhaps even in a > generic way for other sequences using PySequence_GetItem()) > too ? > > I thinking of extending the __getitem__ hook to accept > slice objects. You could then maintain slices to an > object in say a list and apply them to the underlying text > as needed, e.g. > > s = slice(1,4) > l = range(10) > l[s] == [1, 2, 3] I agree that this would be nice to have. I won't have time for it before 2.2 is out, but if you want to give it a crack, I might accept it. > 2. Looking through the code for '==' the optimization for > 'a == a' seems too far down the call stack. Since interning > strings makes this case rather common, I'd suggest to > lift the optimization into the ceval loop (right along side > with the INT op INT optimization). I would agree, except there's also the point of view that since types can override __eq__ to always return false, this optimization should not be done in a way that prevents __eq__ from overriding it. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Tue Oct 30 14:00:13 2001 From: guido@python.org (Guido van Rossum) Date: Tue, 30 Oct 2001 09:00:13 -0500 Subject: [Python-Dev] Slices and "==" optimization In-Reply-To: Your message of "Tue, 30 Oct 2001 11:12:43 +0100." <3BDE7D1B.4850329C@lemburg.com> References: <200110300842.f9U8gCX01628@mira.informatik.hu-berlin.de> <3BDE7D1B.4850329C@lemburg.com> Message-ID: <200110301400.JAA24868@cj20424-a.reston1.va.home.com> > As with all micro-optimizations, the win usually only shows > up in some applications. The one I'm writing uses lot's of > > if tagtype == 'starttag': > if tagtype == 'endtag': > ... > > Since the 'starttag' and 'endtag' strings are interned and > I'm also interning the strings which are stored in tagtype, > I'd like to benefit from the fact that the compare will actually > work as 'is'-compare. However, I don't want to use 'is' because > I consider interning only an optimization and not a feature > of the language. Agreed. > That's why I would like the simple > > if (v == w) return 0; > > integrated into the ceval loop right along the INT-compare > optimization. Maybe this could be done as follows: if (v == w && PyString_CheckExact(v)) return 0; > This may also help in some other situations where objects are > shared. Now *that* is the speculation for which Tim will chop off your head. :-) --Guido van Rossum (home page: http://www.python.org/~guido/) From akuchlin@mems-exchange.org Tue Oct 30 14:48:34 2001 From: akuchlin@mems-exchange.org (Andrew Kuchling) Date: Tue, 30 Oct 2001 09:48:34 -0500 Subject: [Python-Dev] 2.2 article updated Message-ID: <E15yaBu-0003xq-00@ute.cnri.reston.va.us> I've updated the 2.2 article with a section that gives an overview of the type/class changes. At this point the text should be essentially done, except for whatever small changes get into CVS between now and 2.2final. Please take a look and send me any comments: http://www.amk.ca/python/2.2/ (Oops! Michael McLay just sent me a note pointing out that I refer to .__get__() as .get() in a few places. So don't bother reporting that.) If your favorite patch or improvement isn't mentioned, please let me know. Also inform me if you aren't mentioned as a contributor to a patch. BTW, I call PEP 261, "Support for "wide" Unicode characters", not finally accepted and not implemented. Is that correct? Is it going to be finished before 2.2final, or should I simply describe it as partially implemented in the final version of the text? --amk From martin@v.loewis.de Tue Oct 30 17:54:55 2001 From: martin@v.loewis.de (Martin v. Loewis) Date: Tue, 30 Oct 2001 18:54:55 +0100 Subject: [Python-Dev] Slices and "==" optimization In-Reply-To: message from Guido van Rossum on Tue, 30 Oct 2001 09:00:13 -0500 Message-ID: <200110301754.f9UHstt02205@mira.informatik.hu-berlin.de> > > That's why I would like the simple > > > > if (v == w) return 0; > > > > integrated into the ceval loop right along the INT-compare > > optimization. > > Maybe this could be done as follows: > > if (v == w && PyString_CheckExact(v)) return 0; Maybe I'm missing some context here: where is this fragment supposed to go to? into ceval.c:COMPARE_OP? What is the "return 0;" doing then? In any case, I think measurements should show how much speed improvement is gained by taking this short-cut. It sounds nice in theory, but ... I just added the block else if ((v == w) && (oparg == EQ) && PyString_CheckExact(v)) { x = Py_True; Py_INCREF(x); } into the code. In an application that almost exclusively does COMPARE_OPs on identical strings, I got a 30% speed-up. OTOH, this same code caused a 10% slowdown if I converted the "==" into "<>". In a real application, total speed-up will depend on two things: - how many COMPARE_OPs are done in the code? - how many of those compare identical strings for equality? Running the PyXML test suite, I counted 120000 cases where slow_compare was done, and only 700 cases identical strings were compared for equality. Regards, Martin From andymac@bullseye.apana.org.au Tue Oct 30 12:15:28 2001 From: andymac@bullseye.apana.org.au (Andrew MacIntyre) Date: Tue, 30 Oct 2001 23:15:28 +1100 (EDT) Subject: [Python-Dev] OS/2 VAC++ patches Message-ID: <Pine.OS2.4.32.0110302258080.643-100000@central> I have looked at Michael Muller's VAC++ patches, and added comments as followups. In summary: 473749 - has several (minor IMO) stylistic issues (addition of OS/2 #ifdefs). The intent of the changes looks OK, as do most of the actual changes (those in OS/2 VAC specific files or existing OS/2 #ifdefs). 474169 - looks good to go (changes inside existing OS/2 #ifdef) 474500 - looks good to go (isolated to OS/2 specific file) Comments on the stylistic issues? -- Andrew I MacIntyre "These thoughts are mine alone..." E-mail: andymac@bullseye.apana.org.au | Snail: PO Box 370 andymac@pcug.org.au | Belconnen ACT 2616 Web: http://www.andymac.org/ | Australia From fdrake@acm.org Tue Oct 30 18:05:11 2001 From: fdrake@acm.org (Fred L. Drake, Jr.) Date: Tue, 30 Oct 2001 13:05:11 -0500 Subject: [Python-Dev] Slices and "==" optimization In-Reply-To: <200110301754.f9UHstt02205@mira.informatik.hu-berlin.de> References: <200110301754.f9UHstt02205@mira.informatik.hu-berlin.de> Message-ID: <15326.60375.319250.547944@grendel.zope.com> Martin v. Loewis writes: > into the code. In an application that almost exclusively does > COMPARE_OPs on identical strings, I got a 30% speed-up. OTOH, this > same code caused a 10% slowdown if I converted the "==" into "<>". It's very questionable whether we'll find an optimization that works well for any serious variety of applications. Perhaps there's some way to build up a lookup table of fast paths, and have the slow path try that first, but I'm sceptical. > Running the PyXML test suite, I counted 120000 cases where > slow_compare was done, and only 700 cases identical strings were > compared for equality. I wouldn't expect XML applications to gain from such an optimization, since many compares will involve Unicode objects, either to each other or to 8-bit strings. -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> PythonLabs at Zope Corporation From martin@v.loewis.de Tue Oct 30 18:41:10 2001 From: martin@v.loewis.de (Martin v. Loewis) Date: Tue, 30 Oct 2001 19:41:10 +0100 Subject: [Python-Dev] 2.2 article updated Message-ID: <200110301841.f9UIfAA04832@mira.informatik.hu-berlin.de> > BTW, I call PEP 261, "Support for "wide" Unicode characters", not > finally accepted and not implemented. Is that correct? Is it going > to be finished before 2.2final, or should I simply describe it as > partially implemented in the final version of the text? To my knowledge, it is completely implemented. You may mention that --disable-unicode is now also supported. There is atleast one error in the PEP, namely that the define for wide Unicode Py_UNICODE_WIDE (Py_UNICODE_SIZE also exists, but isn't used except to define Py_UNICODE_WIDE). I'm also not sure certain the all codecs handle all surrogate cases correctly in all circumstances, but that is in the realm of bug reports. One further Unicode change: The bullet item "When presented with a Unicode filename on Windows ..." should be extended with "On Unix, the locale's character set is used if nl_langinfo(CODESET) is available." Two new builtins are missing in your text, OverflowWarning and super. HTH, Martin From mmuller@enduden.com Tue Oct 30 18:42:45 2001 From: mmuller@enduden.com (Michael Muller) Date: Tue, 30 Oct 2001 13:42:45 -0500 Subject: [Python-Dev] Re: OS/2 VAC++ patches Message-ID: <222.1004467365.385140.1552565879@lilbastard> Andrew, Thanks for reviewing my patches. I am posting a new patch in response to your comments on *749. Andrew MacIntyre wrote: > I have looked at Michael Muller's VAC++ patches, and added comments as > followups. > > In summary: > 473749 - has several (minor IMO) stylistic issues (addition of OS/2 > #ifdefs). The intent of the changes looks OK, as do most of > the actual changes (those in OS/2 VAC specific files or > existing OS/2 #ifdefs). > 474169 - looks good to go (changes inside existing OS/2 #ifdef) > 474500 - looks good to go (isolated to OS/2 specific file) > > Comments on the stylistic issues? > > -- > Andrew I MacIntyre "These thoughts are mine alone..." > E-mail: andymac@bullseye.apana.org.au | Snail: PO Box 370 > andymac@pcug.org.au | Belconnen ACT 2616 > Web: http://www.andymac.org/ | Australia ============================================================================= michaelMuller = mmuller@enduden.com | http://www.cloud9.net/~proteus ----------------------------------------------------------------------------- There is no concept that is more demeaning to the human spirit than the notion that our freedom must be limited in the interests of our own protection. ============================================================================= From mal@lemburg.com Tue Oct 30 20:07:21 2001 From: mal@lemburg.com (M.-A. Lemburg) Date: Tue, 30 Oct 2001 21:07:21 +0100 Subject: [Python-Dev] Slices and "==" optimization References: <200110301754.f9UHstt02205@mira.informatik.hu-berlin.de> Message-ID: <3BDF0879.D758A1A8@lemburg.com> "Martin v. Loewis" wrote: > > > > That's why I would like the simple > > > > > > if (v == w) return 0; > > > > > > integrated into the ceval loop right along the INT-compare > > > optimization. > > > > Maybe this could be done as follows: > > > > if (v == w && PyString_CheckExact(v)) return 0; > > Maybe I'm missing some context here: where is this fragment supposed > to go to? into ceval.c:COMPARE_OP? What is the "return 0;" doing then? It's pseudo code I made up. The real thing will look very much like it though. > In any case, I think measurements should show how much speed > improvement is gained by taking this short-cut. It sounds nice in > theory, but ... > > I just added the block > > else if ((v == w) && (oparg == EQ) && PyString_CheckExact(v)) { > x = Py_True; > Py_INCREF(x); > } You should first test for oparg == EQ, then for v == w and PyString_CheckExact(). > into the code. In an application that almost exclusively does > COMPARE_OPs on identical strings, I got a 30% speed-up. Nice :-) > OTOH, this > same code caused a 10% slowdown if I converted the "==" into "<>". This sounds like an alignment problem caused by the compiler. 10% speedups/slowdowns can easily be produced by randomly moving about a few cases in the ceval loop. All depends on the platform and compiler though. > In a real application, total speed-up will depend on two things: > - how many COMPARE_OPs are done in the code? > - how many of those compare identical strings for equality? As I explained in my other reply, I have code which does: if variable == 'constant string': ... Since the compiler interns the 'constant string' and I can make sure that variable is interned too (by calling intern()), I can easily take advantage of the optimization without giving away any semantics. If variable happens to be some other constant which is used in a Python module (e.g. some kind of flag or parameter), then it will be interned per-se too. Note that the dictionary implementation relies heavily on the same optimization (interning was added to enhance string compare performance in dict lookups). > Running the PyXML test suite, I counted 120000 cases where > slow_compare was done, and only 700 cases identical strings were > compared for equality. As Fred mentioned, this is probably due to the fact that Unicode objects are not interned. Neither are typical XML tag names; could make a difference... don't know. I believe that PyXML spends most of the time in Python calls and not so much in string compares (and that's what I'm trying to avoid in my XML parser approach ;-). -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Consulting & Company: http://www.egenix.com/ Python Software: http://www.lemburg.com/python/ From jeremy@zope.com Tue Oct 30 20:07:07 2001 From: jeremy@zope.com (Jeremy Hylton) Date: Tue, 30 Oct 2001 15:07:07 -0500 (EST) Subject: [Python-Dev] Future of SSL In-Reply-To: <20011029141844.A1967@lilith.hqd-internal> References: <20011027014211.A11092@lilith.hqd-internal> <200110270226.WAA23767@cj20424-a.reston1.va.home.com> <20011026235527.A2919@trump.amber.org> <200110271556.LAA25312@cj20424-a.reston1.va.home.com> <20011027214357.B6993@trump.amber.org> <3BDB6742.E143BF8B@zope.com> <20011028184024.A1066@lilith.hqd-internal> <3BDD1519.31496FB@lemburg.com> <20011029133640.A1804@lilith.hqd-internal> <200110291306.IAA12932@cj20424-a.reston1.va.home.com> <20011029141844.A1967@lilith.hqd-internal> Message-ID: <15327.2155.83315.318149@slothrop.digicool.com> >>>>> "GH" =3D=3D haering python <Gerhard> writes: GH> On Mon, Oct 29, 2001 at 08:06:28AM -0500, Guido van Rossum GH> wrote: >> > [me, in response to a remark from Marc-Andr=E9] What would be >> > your suggestions? Would you prefer to go in the direction of my >> > original proposal - only providing a SSL API, but not the >> > implementation? >> >> Yes, that's how the current SSL support works -- you need to link >> in openssl. GH> So, as long as there are no actual cryptographic algorithms in GH> the Python source tree, but only hooks for OpenSSL, there's no GH> problem? I don't think the export control regs work that way. Crypto-with-a-hole (that is, an API without the actual crypto implementation) is still considered crypto as far as I know. The regulations are pretty simple these days for Open Source projects. Just send a note to the BXA (Commerce Dept.) with the URL to the source. I wonder if it's wise to do this for Python, since it has an SSL interface (albeit a clunky one). Jeremy From gerhard@bigfoot.de Tue Oct 30 20:28:23 2001 From: gerhard@bigfoot.de (Gerhard =?iso-8859-1?Q?H=E4ring?=) Date: Tue, 30 Oct 2001 21:28:23 +0100 Subject: [Python-Dev] Future of SSL In-Reply-To: <15327.2155.83315.318149@slothrop.digicool.com>; from jeremy@zope.com on Tue, Oct 30, 2001 at 03:07:07PM -0500 References: <20011026235527.A2919@trump.amber.org> <200110271556.LAA25312@cj20424-a.reston1.va.home.com> <20011027214357.B6993@trump.amber.org> <3BDB6742.E143BF8B@zope.com> <20011028184024.A1066@lilith.hqd-internal> <3BDD1519.31496FB@lemburg.com> <20011029133640.A1804@lilith.hqd-internal> <200110291306.IAA12932@cj20424-a.reston1.va.home.com> <20011029141844.A1967@lilith.hqd-internal> <15327.2155.83315.318149@slothrop.digicool.com> Message-ID: <20011030212822.A800@lilith.hqd-internal> On Tue, Oct 30, 2001 at 03:07:07PM -0500, Jeremy Hylton wrote: > GH> So, as long as there are no actual cryptographic algorithms in > GH> the Python source tree, but only hooks for OpenSSL, there's no > GH> problem? > > I don't think the export control regs work that way. > Crypto-with-a-hole (that is, an API without the actual crypto > implementation) is still considered crypto as far as I know. > > The regulations are pretty simple these days for Open Source projects. > Just send a note to the BXA (Commerce Dept.) with the URL to the > source. > > I wonder if it's wise to do this for Python, since it has an SSL > interface (albeit a clunky one). Looks like it would (formally) be required, looking thru this text: http://www.bxa.doc.gov/Encryption/EncryptionRuleOct2K.html I better stop reading crypo law texts now >:-< Gerhard -- mail: gerhard <at> bigfoot <dot> de registered Linux user #64239 web: http://www.cs.fhm.edu/~ifw00065/ OpenPGP public key id 86AB43C0 public key fingerprint: DEC1 1D02 5743 1159 CD20 A4B6 7B22 6575 86AB 43C0 reduce(lambda x,y:x+y,map(lambda x:chr(ord(x)^42),tuple('zS^BED\nX_FOY\x0b'))) From barry@zope.com Tue Oct 30 21:11:50 2001 From: barry@zope.com (Barry A. Warsaw) Date: Tue, 30 Oct 2001 16:11:50 -0500 Subject: [Python-Dev] Future of SSL References: <20011027014211.A11092@lilith.hqd-internal> <200110270226.WAA23767@cj20424-a.reston1.va.home.com> <20011026235527.A2919@trump.amber.org> <200110271556.LAA25312@cj20424-a.reston1.va.home.com> <20011027214357.B6993@trump.amber.org> <3BDB6742.E143BF8B@zope.com> <20011028184024.A1066@lilith.hqd-internal> <3BDD1519.31496FB@lemburg.com> <20011029133640.A1804@lilith.hqd-internal> <200110291306.IAA12932@cj20424-a.reston1.va.home.com> <20011029141844.A1967@lilith.hqd-internal> <15327.2155.83315.318149@slothrop.digicool.com> Message-ID: <15327.6038.463871.492896@anthem.wooz.org> >>>>> "JH" == Jeremy Hylton <jeremy@zope.com> writes: JH> I don't think the export control regs work that way. JH> Crypto-with-a-hole (that is, an API without the actual crypto JH> implementation) is still considered crypto as far as I know. JH> The regulations are pretty simple these days for Open Source JH> projects. Just send a note to the BXA (Commerce Dept.) with JH> the URL to the source. JH> I wonder if it's wise to do this for Python, since it has an JH> SSL interface (albeit a clunky one). Another task for the PSF, eh? -Barry From barry@zope.com Tue Oct 30 21:13:48 2001 From: barry@zope.com (Barry A. Warsaw) Date: Tue, 30 Oct 2001 16:13:48 -0500 Subject: [Python-Dev] Slices and "==" optimization References: <3BDE63F8.541E04B2@lemburg.com> <200110301350.IAA24747@cj20424-a.reston1.va.home.com> Message-ID: <15327.6156.273422.675001@anthem.wooz.org> >>>>> "GvR" == Guido van Rossum <guido@python.org> writes: GvR> I agree that this would be nice to have. I won't have time GvR> for it before 2.2 is out, but if you want to give it a crack, GvR> I might accept it. It's also a new feature, so you'd have to override the feature freeze to add it for 2.2. -Barry From hamish_lawson@yahoo.co.uk Tue Oct 30 21:17:18 2001 From: hamish_lawson@yahoo.co.uk (=?iso-8859-1?q?Hamish=20Lawson?=) Date: Tue, 30 Oct 2001 21:17:18 +0000 (GMT) Subject: [Python-Dev] Misleading error message when multiple arguments for write() Message-ID: <20011030211718.10036.qmail@web11001.mail.yahoo.com> The reference to 'tuple' in the error message below seems misleading. Too many arguments have been passed rather than a tuple. Traceback (most recent call last): File "C:\Temp\demo.py", line 3, in ? sys.stdout.write("a", "b") TypeError: argument must be string or read-only character buffer, not tuple Hamish Lawson ____________________________________________________________ Nokia Game is on again. Go to http://uk.yahoo.com/nokiagame/ and join the new all media adventure before November 3rd. From guido@python.org Tue Oct 30 22:18:52 2001 From: guido@python.org (Guido van Rossum) Date: Tue, 30 Oct 2001 17:18:52 -0500 Subject: [Python-Dev] Misleading error message when multiple arguments for write() In-Reply-To: Your message of "Tue, 30 Oct 2001 21:17:18 GMT." <20011030211718.10036.qmail@web11001.mail.yahoo.com> References: <20011030211718.10036.qmail@web11001.mail.yahoo.com> Message-ID: <200110302218.f9UMIsf09646@odiug.zope.com> > The reference to 'tuple' in the error message below seems misleading. > Too many arguments have been passed rather than a tuple. > > Traceback (most recent call last): > File "C:\Temp\demo.py", line 3, in ? > sys.stdout.write("a", "b") > TypeError: argument must be string or read-only character buffer, not > tuple While you've reached the right audience, and this is indeed a bug(let), python-dev is not the right medium for bug reports. Please submit this to the SourceForge bug tracker, which acts as our collective memory for bugs. --Guido van Rossum (home page: http://www.python.org/~guido/) From jack@oratrix.nl Tue Oct 30 22:25:44 2001 From: jack@oratrix.nl (Jack Jansen) Date: Tue, 30 Oct 2001 23:25:44 +0100 Subject: [Python-Dev] slicing of structseq objects fails In-Reply-To: Message by Russell E Owen <owen@astro.washington.edu> , Tue, 30 Oct 2001 10:02:14 -0800 , <p05100302b8049897fbe0@[128.95.99.44]> Message-ID: <20011030222549.4C74AFB04A@oratrix.oratrix.nl> Python-devvers, Russel Owen reported the following bug, and it turns out to be a general bug, not a MacPython one: slicing the structseq-style tuples returned by time.gmtime() returns a sequence with NULLs in it. Shortly afterwards Python crashes on my SGI. Any takers? Recently, Russell E Owen <owen@astro.washington.edu> said: > The following code fails (G4, MacOS 9.2.1) in Classic MacPython 2.2b1: > > import time > > currUTCTuple= time.gmtime(time.time()) > print "currUTCTuple=%r" % (currUTCTuple,) > print "currUTCTuple[3:6]=%r" % (currUTCTuple[3:6],) > fmtTime = "%s:%02i:%02i" % currUTCTuple[3:6] > print "formatted time = ", fmtTime > > It displays (<NULL>, <NULL>, <NULL>) for currUTCTuple[3:6], > ... -- Jack Jansen | ++++ stop the execution of Mumia Abu-Jamal ++++ Jack.Jansen@oratrix.com | ++++ if you agree copy these lines to your sig ++++ www.cwi.nl/~jack | see http://www.xs4all.nl/~tank/spg-l/sigaction.htm From fredrik@pythonware.com Tue Oct 30 22:57:27 2001 From: fredrik@pythonware.com (Fredrik Lundh) Date: Tue, 30 Oct 2001 23:57:27 +0100 Subject: [Python-Dev] slicing of structseq objects fails References: <20011030222549.4C74AFB04A@oratrix.oratrix.nl> Message-ID: <001b01c16196$7855eb90$ced241d5@hagrid> jack wrote: > > The following code fails (G4, MacOS 9.2.1) in Classic MacPython 2.2b1: > > > > import time > > > > currUTCTuple= time.gmtime(time.time()) > > print "currUTCTuple=%r" % (currUTCTuple,) > > print "currUTCTuple[3:6]=%r" % (currUTCTuple[3:6],) > > fmtTime = "%s:%02i:%02i" % currUTCTuple[3:6] > > print "formatted time = ", fmtTime > > > > It displays (<NULL>, <NULL>, <NULL>) for currUTCTuple[3:6], also note: >>> import time >>> tm = time.gmtime() >>> tm (2001, 10, 30, 22, 54, 16, 1, 303, 0) >>> tm.tm_year 2001 >>> tm[:6] (2001, 10, 30, 22, 54, 16) >>> tm[3:6] (<nil>, <nil>, <nil>) in structseq_slice, for(i = low; i < high; ++i) { PyObject *v = obj->ob_item[i]; Py_INCREF(v); PyTuple_SET_ITEM(np, i, v); } should be: for(i = low; i < high; ++i) { PyObject *v = obj->ob_item[i]; Py_INCREF(v); PyTuple_SET_ITEM(np, i-low, v); } </F> From tim@zope.com Tue Oct 30 22:55:49 2001 From: tim@zope.com (Tim Peters) Date: Tue, 30 Oct 2001 17:55:49 -0500 Subject: [Python-Dev] Slices and "==" optimization In-Reply-To: <3BDF0879.D758A1A8@lemburg.com> Message-ID: <BIEJKCLHCIOIHAGOKOLHMEKCCJAA.tim@zope.com> [MvL] >> OTOH, this same code caused a 10% slowdown if I converted the "==" into >> "<>". [MAL] > This sounds like an alignment problem caused by the compiler. > 10% speedups/slowdowns can easily be produced by randomly moving > about a few cases in the ceval loop. All depends on the platform > and compiler though. I expect a more obvious cause is at work: every one of Martin's "<>" compares was slowed by the new batch of failing "is it an EQ and are these strings and are they identical?" tests and branches. Fast paths lose when when they don't win -- they aren't neutral then. Indeed, we could speed your particular app by getting rid of the int-compare fast path already there (but I expect that one is a *net* win across apps, so don't want to get rid of it <wink>). >> Running the PyXML test suite, I counted 120000 cases where >> slow_compare was done, and only 700 cases identical strings were >> compared for equality. > As Fred mentioned, this is probably due to the fact that > Unicode objects are not interned. Neither are typical > XML tag names; could make a difference... don't know. Whatever it's due to, it means the fast-path code cost extra cycles in the 119300 (of 120000) cases it didn't pay off. If it costs F extra cycles when it fails, and saves S cycles when it succeeds, the question for this test is whether 119300*F < 700*S. S has to about about 170X larger than F for a net win in this case, else there's a net loss. From tim@zope.com Tue Oct 30 23:51:11 2001 From: tim@zope.com (Tim Peters) Date: Tue, 30 Oct 2001 18:51:11 -0500 Subject: [Python-Dev] slicing of structseq objects fails In-Reply-To: <20011030222549.4C74AFB04A@oratrix.oratrix.nl> Message-ID: <BIEJKCLHCIOIHAGOKOLHIEKECJAA.tim@zope.com> [Jack Jansen] > Python-devvers, > Russel Owen reported the following bug, and it turns out to be a > general bug, not a MacPython one: slicing the structseq-style tuples > returned by time.gmtime() returns a sequence with NULLs in it. Shortly > afterwards Python crashes on my SGI. > > Any takers? Sure, it was a shallow bug in the new structseq code and has been fixed in current CVS. Next time, how about a bug report on SourceForge instead? From tim@zope.com Tue Oct 30 23:54:07 2001 From: tim@zope.com (Tim Peters) Date: Tue, 30 Oct 2001 18:54:07 -0500 Subject: [Python-Dev] slicing of structseq objects fails In-Reply-To: <001b01c16196$7855eb90$ced241d5@hagrid> Message-ID: <BIEJKCLHCIOIHAGOKOLHCEKFCJAA.tim@zope.com> [/F] > in structseq_slice, > > for(i = low; i < high; ++i) { > PyObject *v = obj->ob_item[i]; > Py_INCREF(v); > PyTuple_SET_ITEM(np, i, v); > } > > should be: > > for(i = low; i < high; ++i) { > PyObject *v = obj->ob_item[i]; > Py_INCREF(v); > PyTuple_SET_ITEM(np, i-low, v); > } Bingo. First Jack reports a bug here, then /F goes to the trouble of copying in before-and-after code instead of just checking in the fix. Does everyone really hate SourceForge that much <wink>? From fredrik@pythonware.com Wed Oct 31 08:04:05 2001 From: fredrik@pythonware.com (Fredrik Lundh) Date: Wed, 31 Oct 2001 09:04:05 +0100 Subject: [Python-Dev] slicing of structseq objects fails References: <BIEJKCLHCIOIHAGOKOLHCEKFCJAA.tim@zope.com> Message-ID: <00c401c161e2$ba34f320$ced241d5@hagrid> tim wrote: > Bingo. First Jack reports a bug here, then /F goes to the trouble of > copying in before-and-after code instead of just checking in the fix. I didn't have enough time to rebuild and validate the patch, and I didn't want to risk checking anything in that didn't compile ;-) </F> From mal@lemburg.com Wed Oct 31 08:08:26 2001 From: mal@lemburg.com (M.-A. Lemburg) Date: Wed, 31 Oct 2001 09:08:26 +0100 Subject: [Python-Dev] Slices and "==" optimization References: <BIEJKCLHCIOIHAGOKOLHMEKCCJAA.tim@zope.com> Message-ID: <3BDFB17A.78584E7C@lemburg.com> Tim Peters wrote: > > >> Running the PyXML test suite, I counted 120000 cases where > >> slow_compare was done, and only 700 cases identical strings were > >> compared for equality. > > > As Fred mentioned, this is probably due to the fact that > > Unicode objects are not interned. Neither are typical > > XML tag names; could make a difference... don't know. > > Whatever it's due to, it means the fast-path code cost extra cycles in the > 119300 (of 120000) cases it didn't pay off. If it costs F extra cycles when > it fails, and saves S cycles when it succeeds, the question for this test is > whether 119300*F < 700*S. S has to about about 170X larger than F for a net > win in this case, else there's a net loss. I'd be interested in the overall performance change in the PyXML test suite run. If you look at the code that gets executed for the checking the fast path criteria down in slow_compare, then your equation doesn't look that bad anymore. I'll run pybench on this and post the results here. -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Consulting & Company: http://www.egenix.com/ Python Software: http://www.lemburg.com/python/ From jack@oratrix.nl Wed Oct 31 10:42:47 2001 From: jack@oratrix.nl (Jack Jansen) Date: Wed, 31 Oct 2001 11:42:47 +0100 Subject: [Python-Dev] slicing of structseq objects fails In-Reply-To: Message by "Tim Peters" <tim@zope.com> , Tue, 30 Oct 2001 18:54:07 -0500 , <BIEJKCLHCIOIHAGOKOLHCEKFCJAA.tim@zope.com> Message-ID: <20011031104248.F342E303181@snelboot.oratrix.nl> > Bingo. First Jack reports a bug here, [...] Ok, Tim, sorry! You're right, of course, that I should have submitted the bug, it was just that I was really busy with other things, and cc-ing python-dev was a lot quicker than first replying to the message to the poster and pythonmac-sig, finding out who owns structseq, filing the bug report and telling the poster about the id (I've found that pointing people to the sourceforge bug reporter to report the bug themselves means the bug is lost in >50% of the cases). But saving my time shouldn't be an excuse for wasting everyone else's. Sorry, -- Jack Jansen | ++++ stop the execution of Mumia Abu-Jamal ++++ Jack.Jansen@oratrix.com | ++++ if you agree copy these lines to your sig ++++ www.cwi.nl/~jack | see http://www.xs4all.nl/~tank/spg-l/sigaction.htm From jim@interet.com Wed Oct 31 16:44:34 2001 From: jim@interet.com (James C. Ahlstrom) Date: Wed, 31 Oct 2001 11:44:34 -0500 Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives References: <3BD9B5EC.119319A5@interet.com> <200110262034.QAA20802@cj20424-a.reston1.va.home.com> <3BDD6588.7CFEA7D6@interet.com> <200110291458.JAA13956@cj20424-a.reston1.va.home.com> <3BDD91CC.9F0ECB07@interet.com> <200110291739.MAA15773@cj20424-a.reston1.va.home.com> Message-ID: <3BE02A72.372AE487@interet.com> Guido van Rossum wrote: > > > OK. I propose that there is one name added to sys.path, and the > > file name is "python%s%s.zip" % (sys.version[0], sys.version[2]). > Sounds good to me. Somebody please update the PEP. The PEP is updated. Did we decide to allow import of *.py? JimA From jack@oratrix.nl Wed Oct 31 16:55:23 2001 From: jack@oratrix.nl (Jack Jansen) Date: Wed, 31 Oct 2001 17:55:23 +0100 Subject: [Python-Dev] Patch submitted for cross-platform newline support Message-ID: <20011031165523.85D36303181@snelboot.oratrix.nl> Hi folks, I've just submitted a patch that allows Python to do cross-platform newline handling, #476814. This sort-of happened after the promised PEP didn't materialise, and there was discussion about this among myself, Guido and the pythonmac-sig, and at some point Guido said "Ok, code it!". The patch has been tested on SGI Irix and in MacPython only. I'd like people to have a look at this. There's one thing I don't like, and that's the changes I had to make to the parser sourcefiles. They're needed, but I'm open to suggestions as to a better way to do this. And of course I'm open to any suggestions. I'm also interested in discussing whether a patch like this is appropriate while we're in beta. On the one hand I would say it is, because the feature is disabled by default. On the other hand there are changes (albeit mainly cosmetic ones) in a large number of places. Another argument for allowing this even while in beta is that I really really want it for Mac OS X (but this might not be a very strong argument, I guess:-). Here's the summary of the patch: This patch enables Python to interpret all known newline conventions, CR, LF or CRLF, on all platforms. This support is enabled by configuring with --with-universal-newlines (so by default it is off, and everything should behave as usual). With universal newline support enabled two things happen: - When importing or otherwise parsing .py files any newline convention is accepted. - Python code can pass a new "t" mode parameter to open() which reads files with any newline convention. "t" cannot be combined with any other mode flags like "w" or "+", for obvious reasons. File objects have a new attribute "newlines" which contains the type of newlines encountered in the file (or None when no newline has been seen, or "mixed" if there were various types of newlines). Also included is a test script which tests both file I/O and parsing. -- Jack Jansen | ++++ stop the execution of Mumia Abu-Jamal ++++ Jack.Jansen@oratrix.com | ++++ if you agree copy these lines to your sig ++++ www.cwi.nl/~jack | ++++ see http://www.xs4all.nl/~tank/ ++++ From fredrik@pythonware.com Wed Oct 31 18:19:26 2001 From: fredrik@pythonware.com (Fredrik Lundh) Date: Wed, 31 Oct 2001 19:19:26 +0100 Subject: [Python-Dev] Patch submitted for cross-platform newline support References: <20011031165523.85D36303181@snelboot.oratrix.nl> Message-ID: <007601c16238$9e662730$ced241d5@hagrid> Jack wrote: > - Python code can pass a new "t" mode parameter to open() which reads > files with any newline convention. "t" cannot be combined with any > other mode flags like "w" or "+", for obvious reasons. so "t" can be both a mode (like "r" and "w") and a mode modifier (like "t" and "b")? could be confusing. </F> From gward@python.net Wed Oct 31 19:14:10 2001 From: gward@python.net (Greg Ward) Date: Wed, 31 Oct 2001 14:14:10 -0500 Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives In-Reply-To: <3BE02A72.372AE487@interet.com> References: <3BD9B5EC.119319A5@interet.com> <200110262034.QAA20802@cj20424-a.reston1.va.home.com> <3BDD6588.7CFEA7D6@interet.com> <200110291458.JAA13956@cj20424-a.reston1.va.home.com> <3BDD91CC.9F0ECB07@interet.com> <200110291739.MAA15773@cj20424-a.reston1.va.home.com> <3BE02A72.372AE487@interet.com> Message-ID: <20011031141410.A12872@gerg.ca> On 31 October 2001, James C. Ahlstrom said: > Did we decide to allow import of *.py? Here's an idea that would apply to importing *.py whether inside ZIP files or not: if unable to write the .pyc, write a warning to stderr. That'll be useful to people who inadvertently put .py files without .pyc in a ZIP file, and to people who try to import .py files from a directory they can't write to, etc. Now that Python has a warning framework, warnings like unable to write /usr/lib/python2.2/site-packages/foo.pyc: permission denied or unable to write /usr/lib/python2.2/site-packages/foo.zip/foo.pyc: can't write to ZIP file sound like useful warnings. Greg -- Greg Ward - programmer-at-large gward@python.net http://starship.python.net/~gward/ Save energy: be apathetic. From skip@pobox.com (Skip Montanaro) Wed Oct 31 19:15:34 2001 From: skip@pobox.com (Skip Montanaro) (Skip Montanaro) Date: Wed, 31 Oct 2001 13:15:34 -0600 Subject: [Python-Dev] statcache.lstat? Message-ID: <15328.19926.368998.631005@beluga.mojam.com> Does this (untested) code make sense as a possible addition to statcache? One obvious problem I see is that if stat(path) had been called before lstat(path), you'd pull the wrong info out of the cache. You could get around this by maintaining separate stat and lstat caches or caching a different key (e.g. (path, "l")) for the lstat variety. If this is deemed worthwhile I'll make it work and submit a patch to SF. I wanted to check first to make sure I wasn't missing something obvious that would keep it from ever working. Skip *** /tmp/skip/statcache.py.~1.11~fVASKs Wed Oct 31 13:10:22 2001 --- /tmp/skip/statcache.pyfVAfUy Wed Oct 31 13:10:22 2001 *************** *** 6,12 **** import os as _os from stat import * ! __all__ = ["stat","reset","forget","forget_prefix","forget_dir", "forget_except_prefix","isdir"] # The cache. Keys are pathnames, values are os.stat outcomes. --- 6,12 ---- import os as _os from stat import * ! __all__ = ["stat","lstat","reset","forget","forget_prefix","forget_dir", "forget_except_prefix","isdir"] # The cache. Keys are pathnames, values are os.stat outcomes. *************** *** 22,27 **** --- 22,37 ---- if ret is None: cache[path] = ret = _os.stat(path) return ret + + if hasattr(_os, "lstat"): + def lstat(path): + """Lstat a file, possibly out of the cache.""" + ret = cache.get(path, None) + if ret is None: + cache[path] = ret = _os.lstat(path) + return ret + else: + lstat = stat def reset(): """Clear the cache.""" From fredrik@pythonware.com Wed Oct 31 19:37:18 2001 From: fredrik@pythonware.com (Fredrik Lundh) Date: Wed, 31 Oct 2001 20:37:18 +0100 Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives References: <3BD9B5EC.119319A5@interet.com> <200110262034.QAA20802@cj20424-a.reston1.va.home.com> <3BDD6588.7CFEA7D6@interet.com> <200110291458.JAA13956@cj20424-a.reston1.va.home.com> <3BDD91CC.9F0ECB07@interet.com> <200110291739.MAA15773@cj20424-a.reston1.va.home.com> <3BE02A72.372AE487@interet.com> <20011031141410.A12872@gerg.ca> Message-ID: <004701c16244$27bd9a80$ced241d5@hagrid> Greg wrote: > > > Did we decide to allow import of *.py? > > Here's an idea that would apply to importing *.py whether inside ZIP > files or not: if unable to write the .pyc, write a warning to stderr. > That'll be useful to people who inadvertently put .py files without .pyc > in a ZIP file can anyone explain why I shouldn't be allowed to ship, say, PIL's Python code in a version-independent ZIP file? </F> From jeremy@zope.com Wed Oct 31 19:43:37 2001 From: jeremy@zope.com (Jeremy Hylton) Date: Wed, 31 Oct 2001 14:43:37 -0500 (EST) Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives In-Reply-To: <20011031141410.A12872@gerg.ca> References: <3BD9B5EC.119319A5@interet.com> <200110262034.QAA20802@cj20424-a.reston1.va.home.com> <3BDD6588.7CFEA7D6@interet.com> <200110291458.JAA13956@cj20424-a.reston1.va.home.com> <3BDD91CC.9F0ECB07@interet.com> <200110291739.MAA15773@cj20424-a.reston1.va.home.com> <3BE02A72.372AE487@interet.com> <20011031141410.A12872@gerg.ca> Message-ID: <15328.21609.836727.605337@slothrop.digicool.com> >>>>> "GW" == Greg Ward <gward@python.net> writes: GW> Here's an idea that would apply to importing *.py whether inside GW> ZIP files or not: if unable to write the .pyc, write a warning GW> to stderr. This is exactly what the interpreter does when run with -v. If you load a .py file and can't write the .pyc -- say you don't have write permission in the directory -- it prints a warning to stderr. +1 on ZIP import doing the same thing Jeremy From jeremy@zope.com Wed Oct 31 19:48:29 2001 From: jeremy@zope.com (Jeremy Hylton) Date: Wed, 31 Oct 2001 14:48:29 -0500 (EST) Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives In-Reply-To: <004701c16244$27bd9a80$ced241d5@hagrid> References: <3BD9B5EC.119319A5@interet.com> <200110262034.QAA20802@cj20424-a.reston1.va.home.com> <3BDD6588.7CFEA7D6@interet.com> <200110291458.JAA13956@cj20424-a.reston1.va.home.com> <3BDD91CC.9F0ECB07@interet.com> <200110291739.MAA15773@cj20424-a.reston1.va.home.com> <3BE02A72.372AE487@interet.com> <20011031141410.A12872@gerg.ca> <004701c16244$27bd9a80$ced241d5@hagrid> Message-ID: <15328.21901.6102.308855@slothrop.digicool.com> >>>>> "FL" == Fredrik Lundh <fredrik@pythonware.com> writes: FL> can anyone explain why I shouldn't be allowed to ship, say, FL> PIL's Python code in a version-independent ZIP file? Excellent point! We ought to collect some requirements/goals for the ZIP import stuff. The current draft of the PEP is a little thin on motivation. Requirement: A ZIP archive be usable any version of Python that supports ZIP-based imports. It might also be nice to have a imputil- or ihook-based importer that works with older versions of Python. Jeremy From tim@zope.com Wed Oct 31 20:14:33 2001 From: tim@zope.com (Tim Peters) Date: Wed, 31 Oct 2001 15:14:33 -0500 Subject: [Python-Dev] Patch submitted for cross-platform newline support In-Reply-To: <007601c16238$9e662730$ced241d5@hagrid> Message-ID: <BIEJKCLHCIOIHAGOKOLHEEMLCJAA.tim@zope.com> [Jack Jansen] > - Python code can pass a new "t" mode parameter to open() which reads > files with any newline convention. "t" cannot be combined with any > other mode flags like "w" or "+", for obvious reasons. [/F] > so "t" can be both a mode (like "r" and "w") and a mode modifier > (like "t" and "b")? could be confusing. Well, the "t" modifier isn't std; does anyone other than Microsoft support it? MS also gives meaning to non-std "c" and "n" modifiers. Luckily, Python doesn't define which modes or modifiers it accepts <wink>. From gmcm@hypernet.com Wed Oct 31 21:04:31 2001 From: gmcm@hypernet.com (Gordon McMillan) Date: Wed, 31 Oct 2001 16:04:31 -0500 Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives In-Reply-To: <15328.21901.6102.308855@slothrop.digicool.com> References: <004701c16244$27bd9a80$ced241d5@hagrid> Message-ID: <3BE0210F.12966.4496B695@localhost> > >>>>> "FL" == Fredrik Lundh <fredrik@pythonware.com> writes: > > FL> can anyone explain why I shouldn't be allowed to ship, say, > FL> PIL's Python code in a version-independent ZIP file? Jeremy Hylton: > Excellent point! Being the permissive sort, I'd say you should be allowed. But I think it would be foolish. > We ought to collect some requirements/goals for the ZIP import > stuff. The current draft of the PEP is a little thin on > motivation. Speed, ease of deployment, minimal disk space. MacPython does it, Jython does it, Installer's been doing it since '98 (although it doesn't use zip format archives). > Requirement: A ZIP archive be usable any version of Python that > supports ZIP-based imports. Hmm. You want .pyc's with every known magic number (there goes disk space)? Or just source and throw away the .pyc? Or maybe insert into the zip file? Either way, there goes speed. Besides, as I noted, and Guido concurred, we need some way to handle extension modules "inside" packages, since they can't go inside the zip. On Windows, extensions are only usable by one version of Python. This is silliness. I have clients with versions 1.52 to 2.1. I have a fair amount of code I reuse, and I keep it version- independent. I have had to change something in it for every Python version to keep it version independent. > It might also be nice to have a imputil- or ihook-based importer > that works with older versions of Python. There's at least one imputil-based one that works with zipfile. - Gordon From skip@pobox.com (Skip Montanaro) Wed Oct 31 21:11:12 2001 From: skip@pobox.com (Skip Montanaro) (Skip Montanaro) Date: Wed, 31 Oct 2001 15:11:12 -0600 Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives In-Reply-To: <3BE0210F.12966.4496B695@localhost> References: <004701c16244$27bd9a80$ced241d5@hagrid> <3BE0210F.12966.4496B695@localhost> Message-ID: <15328.26864.350472.491216@beluga.mojam.com> >> Requirement: A ZIP archive be usable any version of Python that >> supports ZIP-based imports. Gordon> Hmm. You want .pyc's with every known magic number (there goes Gordon> disk space)? Or just source and throw away the .pyc? Or maybe Gordon> insert into the zip file? Either way, there goes speed. Can't zip files be treated conceptually as directories? If I import a py-based module from a zip archive I see no particular reason the byte-compiled file can't be added to the archive (conceptually, written to the directory), speeding up the import the next time. Is this not possible? Skip From mal@lemburg.com Wed Oct 31 21:12:02 2001 From: mal@lemburg.com (M.-A. Lemburg) Date: Wed, 31 Oct 2001 22:12:02 +0100 Subject: [Python-Dev] pickle faster in 2.2 ? Message-ID: <3BE06922.5C43E09F@lemburg.com> While hacking on an XML pickler, I found that pickle.py got nearly twice as fast in 2.2 comparing to 2.1 and 2.0. The code in pickle.py doesn't seem to have changed much. Anybody know where that speedup came from ? Can somebody on another (non-Linux) system please verify this. Thanks, -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Consulting & Company: http://www.egenix.com/ Python Software: http://www.lemburg.com/python/ From guido@python.org Wed Oct 31 21:20:23 2001 From: guido@python.org (Guido van Rossum) Date: Wed, 31 Oct 2001 16:20:23 -0500 Subject: [Python-Dev] statcache.lstat? In-Reply-To: Your message of "Wed, 31 Oct 2001 13:15:34 CST." <15328.19926.368998.631005@beluga.mojam.com> References: <15328.19926.368998.631005@beluga.mojam.com> Message-ID: <200110312120.QAA32561@cj20424-a.reston1.va.home.com> > Does this (untested) code make sense as a possible addition to statcache? For me, statcache was a failed experiment. What's the use you have in mind? Why can't you use os.lstat() directly? > One obvious problem I see is that if stat(path) had been called before > lstat(path), you'd pull the wrong info out of the cache. You could get > around this by maintaining separate stat and lstat caches or caching a > different key (e.g. (path, "l")) for the lstat variety. Definitely use two caches or a separate key. --Guido van Rossum (home page: http://www.python.org/~guido/) From mal@lemburg.com Wed Oct 31 21:22:46 2001 From: mal@lemburg.com (M.-A. Lemburg) Date: Wed, 31 Oct 2001 22:22:46 +0100 Subject: [Python-Dev] pickle faster in 2.2 ? References: <3BE06922.5C43E09F@lemburg.com> Message-ID: <3BE06BA6.59EE23E@lemburg.com> "M.-A. Lemburg" wrote: > > While hacking on an XML pickler, I found that pickle.py got nearly > twice as fast in 2.2 comparing to 2.1 and 2.0. > > The code in pickle.py doesn't seem to have changed much. Anybody > know where that speedup came from ? Can somebody on another > (non-Linux) system please verify this. Nevermind; it was the change in pickle.py to use cStringIO instead of StringIO that caused the difference. -- Marc-Andre Lemburg CEO eGenix.com Software GmbH ______________________________________________________________________ Consulting & Company: http://www.egenix.com/ Python Software: http://www.lemburg.com/python/ From guido@python.org Wed Oct 31 21:31:53 2001 From: guido@python.org (Guido van Rossum) Date: Wed, 31 Oct 2001 16:31:53 -0500 Subject: [Python-Dev] pickle faster in 2.2 ? In-Reply-To: Your message of "Wed, 31 Oct 2001 22:12:02 +0100." <3BE06922.5C43E09F@lemburg.com> References: <3BE06922.5C43E09F@lemburg.com> Message-ID: <200110312131.QAA32734@cj20424-a.reston1.va.home.com> > While hacking on an XML pickler, I found that pickle.py got nearly > twice as fast in 2.2 comparing to 2.1 and 2.0. What benchmark? > The code in pickle.py doesn't seem to have changed much. Anybody > know where that speedup came from ? Can somebody on another > (non-Linux) system please verify this. I believe DOM nodes are now new-style classes, for better or for worse (it might create problems when combining with classic mixins). Could that explain it? --Guido van Rossum (home page: http://www.python.org/~guido/) From gmcm@hypernet.com Wed Oct 31 22:31:02 2001 From: gmcm@hypernet.com (Gordon McMillan) Date: Wed, 31 Oct 2001 17:31:02 -0500 Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives In-Reply-To: <15328.26864.350472.491216@beluga.mojam.com> References: <3BE0210F.12966.4496B695@localhost> Message-ID: <3BE03556.11078.44E5ECD7@localhost> Jeremy: > >> Requirement: A ZIP archive be usable any version of Python > that >> supports ZIP-based imports. > > Gordon> Hmm. You want .pyc's with every known magic number > (there goes Gordon> disk space)? Or just source and throw > away the .pyc? Or maybe Gordon> insert into the zip file? > Either way, there goes speed. Skip: > Can't zip files be treated conceptually as directories? If I > import a py-based module from a zip archive I see no particular > reason the byte-compiled file can't be added to the archive > (conceptually, written to the directory), speeding up the import > the next time. Is this not possible? Zip files are normally unpacked to the filesystem and rebuilt when modified. If this feature is desired, I think MetaKit would be a much better file format (I have used MetaKit as a pyc archive so I could replace individual modules). But unless you want to big-boy SQL servers as "archives", you're going to have trouble with multi-user. To combine this with Jeremy's requirement, we can't put the .pyc's alongside the py's anyway, we need some (disappearing) node in the hierarchy to hold the byte code version (magic number). A bit more trickery for import.c. We can meet a real need as long as we don't put tail fins and heavy chrome all over a Yugo (zip files). They're easy to produce & ship. I don't think we should disallow .py files, but I think 99% of their use will be version specific .pyc's only (and many of those to users who think "traceback, innermost last" is Aramaic). It's nothing compared to the effort of dealing with the binaries across versions (or even the std lib across versions, sometimes). OTOH, I'm worried that the extension-modules-inside- packages requirement is going to break import.c's back. There are many undocumented features of the implementation of import that are relied on (such as xml.__init__'s trick, or extentions modules inside packages). Heck, we might even have to clarify the rules. doom-and-gloom-and-goblins-ly y'rs - Gordon From skip@pobox.com (Skip Montanaro) Wed Oct 31 22:34:17 2001 From: skip@pobox.com (Skip Montanaro) (Skip Montanaro) Date: Wed, 31 Oct 2001 16:34:17 -0600 Subject: [Python-Dev] statcache.lstat? In-Reply-To: <200110312120.QAA32561@cj20424-a.reston1.va.home.com> References: <15328.19926.368998.631005@beluga.mojam.com> <200110312120.QAA32561@cj20424-a.reston1.va.home.com> Message-ID: <15328.31849.14360.58802@beluga.mojam.com> >> Does this (untested) code make sense as a possible addition to >> statcache? Guido> For me, statcache was a failed experiment. What's the use you Guido> have in mind? Why can't you use os.lstat() directly? I can use os.lstat (or os.stat) directly. I'm working on a file selector widget written in Python (and PyGtk). As people traverse the directory tree, it seems to make sense to cache the stat results. If statcache is indeed a failed experiment, perhaps it should be deprecated. Skip From guido@python.org Wed Oct 31 22:38:03 2001 From: guido@python.org (Guido van Rossum) Date: Wed, 31 Oct 2001 17:38:03 -0500 Subject: [Python-Dev] statcache.lstat? In-Reply-To: Your message of "Wed, 31 Oct 2001 16:34:17 CST." <15328.31849.14360.58802@beluga.mojam.com> References: <15328.19926.368998.631005@beluga.mojam.com> <200110312120.QAA32561@cj20424-a.reston1.va.home.com> <15328.31849.14360.58802@beluga.mojam.com> Message-ID: <200110312238.RAA01093@cj20424-a.reston1.va.home.com> > Guido> For me, statcache was a failed experiment. What's the use you > Guido> have in mind? Why can't you use os.lstat() directly? > > I can use os.lstat (or os.stat) directly. I'm working on a file selector > widget written in Python (and PyGtk). As people traverse the directory > tree, it seems to make sense to cache the stat results. But why bother? And why not let them see changes in the filesystem? > If statcache is indeed a failed experiment, perhaps it should be > deprecated. Fine with me. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Wed Oct 31 22:41:44 2001 From: guido@python.org (Guido van Rossum) Date: Wed, 31 Oct 2001 17:41:44 -0500 Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives In-Reply-To: Your message of "Wed, 31 Oct 2001 17:31:02 EST." <3BE03556.11078.44E5ECD7@localhost> References: <3BE0210F.12966.4496B695@localhost> <3BE03556.11078.44E5ECD7@localhost> Message-ID: <200110312241.RAA01122@cj20424-a.reston1.va.home.com> - Writing stuff back to .zip files is totally the wrong approach. - I don't care about having .so files inside packages *in zipfiles*. - I'm not sure I care about having .so files inside packages on the filesystem; they are useful in Zope, but for very hackish reasons. - If the zip file has the .py file but no .pyc or the wrong .pyc, tant pis. Let it be slower. (But if it has the .pyo, use that.) --Guido van Rossum (home page: http://www.python.org/~guido/) From just@letterror.com Wed Oct 31 23:07:35 2001 From: just@letterror.com (Just van Rossum) Date: Thu, 1 Nov 2001 00:07:35 +0100 Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives In-Reply-To: <200110312241.RAA01122@cj20424-a.reston1.va.home.com> Message-ID: <20011101000740-r01010800-a1f25692-0910-010c@10.0.0.23> Guido van Rossum wrote: > - I'm not sure I care about having .so files inside packages on the > filesystem; they are useful in Zope, but for very hackish reasons. Why? If I write a package which is mostly in Python, it feels very natural to put the C extensions also in the package. Just From jack@oratrix.nl Wed Oct 31 23:14:58 2001 From: jack@oratrix.nl (Jack Jansen) Date: Thu, 01 Nov 2001 00:14:58 +0100 Subject: [Python-Dev] Patch submitted for cross-platform newline support In-Reply-To: Message by "Tim Peters" <tim@zope.com> , Wed, 31 Oct 2001 15:14:33 -0500 , <BIEJKCLHCIOIHAGOKOLHEEMLCJAA.tim@zope.com> Message-ID: <20011031231503.092DB151ADB@oratrix.oratrix.nl> Recently, "Tim Peters" <tim@zope.com> said: > Well, the "t" modifier isn't std; does anyone other than Microsoft support > it? MS also gives meaning to non-std "c" and "n" modifiers. Luckily, > Python doesn't define which modes or modifiers it accepts <wink>. Oops, big oops! I picked the "t" because it was intended _not_ to be supported on any platform: my code eats it, and turns on the universal newline feature on the file object. What does "t" do on Windows? What would be a better choice? -- Jack Jansen | ++++ stop the execution of Mumia Abu-Jamal ++++ Jack.Jansen@oratrix.com | ++++ if you agree copy these lines to your sig ++++ www.cwi.nl/~jack | see http://www.xs4all.nl/~tank/spg-l/sigaction.htm From guido@python.org Wed Oct 31 23:38:55 2001 From: guido@python.org (Guido van Rossum) Date: Wed, 31 Oct 2001 18:38:55 -0500 Subject: [Python-Dev] PEP 273: Import Modules from Zip Archives In-Reply-To: Your message of "Thu, 01 Nov 2001 00:07:35 +0100." <20011101000740-r01010800-a1f25692-0910-010c@10.0.0.23> References: <20011101000740-r01010800-a1f25692-0910-010c@10.0.0.23> Message-ID: <200110312338.SAA01368@cj20424-a.reston1.va.home.com> > Guido van Rossum wrote: > > > - I'm not sure I care about having .so files inside packages on the > > filesystem; they are useful in Zope, but for very hackish reasons. > > Why? If I write a package which is mostly in Python, it feels very > natural to put the C extensions also in the package. > > Just Yes, it does, and as long as it works, I have no problem with that. Distutils supports this too, AFAIK. But if there are mechanical problems with making it work (zip files are a good example of that) I don't see why we should torture ourselves to get it to work when simpler solutions exist (such as putting the extension at the top level under a private name with the package name as a prefix). BTW, the hackish reasons I referred to are this: Zope often wants to *replace* existing extensions with its own versions, and places e.g. its own cPickle.so insize a package to force the import even if cPickle is built-in statically. But I'm not sure that that works if there's a toplevel cPickle.so which has already been imported; it may work on some systems but fail on others, depending on the shared library architecture (often one of the most hackish parts of user-space OS support). --Guido van Rossum (home page: http://www.python.org/~guido/) From tim@zope.com Wed Oct 31 23:44:44 2001 From: tim@zope.com (Tim Peters) Date: Wed, 31 Oct 2001 18:44:44 -0500 Subject: [Python-Dev] Patch submitted for cross-platform newline support In-Reply-To: <20011031231503.092DB151ADB@oratrix.oratrix.nl> Message-ID: <BIEJKCLHCIOIHAGOKOLHKENJCJAA.tim@zope.com> [Jack Jansen] > Oops, big oops! I picked the "t" because it was intended _not_ to be > supported on any platform: my code eats it, and turns on the universal > newline feature on the file object. What does "t" do on Windows? MS treats "t" as an explicit way of saying "not b" <wink>, i.e. text mode. The MS libraries can be configured to use "b" as the default, and then you need "t" to force text mode (when needed). > What would be a better choice? I don't know. MS adds t, c and n as extensions to C's fopen() gimmicks, and there's really no way to guess what the union of all other platforms may do. Python (before your patch) passes on *whatever* mode string the user supplies, without even peeking at it. That's a mixed blessing: I've been very happy to be able to pass "c" on MS (which triggers a "commit" mode that vastly increases the chances file output winds up on disk after a crash); OTOH, across platforms the kinds of error msgs we get for malformed mode strings aren't very good: >>> f = file('oops', 'br') Traceback (most recent call last): File "<stdin>", line 1, in ? IOError: [Errno 0] Error: 'oops' >>> Radical idea: don't do anything to turn on "universal newlines" -- say it's just what "text mode" means in Python. Then you only have to worry about picking a letter to turn it off <wink>. From guido@python.org Wed Oct 31 23:48:20 2001 From: guido@python.org (Guido van Rossum) Date: Wed, 31 Oct 2001 18:48:20 -0500 Subject: [Python-Dev] Patch submitted for cross-platform newline support In-Reply-To: Your message of "Wed, 31 Oct 2001 18:44:44 EST." <BIEJKCLHCIOIHAGOKOLHKENJCJAA.tim@zope.com> References: <BIEJKCLHCIOIHAGOKOLHKENJCJAA.tim@zope.com> Message-ID: <200110312348.SAA01457@cj20424-a.reston1.va.home.com> > Radical idea: don't do anything to turn on "universal newlines" -- say it's > just what "text mode" means in Python. Then you only have to worry about > picking a letter to turn it off <wink>. That would be my choice. KISS. --Guido van Rossum (home page: http://www.python.org/~guido/)