Mailman 3 May 2002 - Python-Dev

RE: [Python-Dev] Those import related syntax errors again...
by Samuele Pedroni 23 Feb '05

23 Feb '05

Hi. [Mark Hammond] > The point isn't about my suffering as such. The point is more that > python-dev owns a tiny amount of the code out there, and I don't believe we > should put Python's users through this. > > Sure - I would be happy to "upgrade" all the win32all code, no problem. I > am also happy to live in the bleeding edge and take some pain that will > cause. > > The issue is simply the user base, and giving Python a reputation of not > being able to painlessly upgrade even dot revisions. I agree with all this. [As I imagined explicit syntax did not catch up and would require lot of discussions.] [GvR] > > Another way is to use special rules > > (similar to those for class defs), e.g. having > > > > <frag> > > y=3 > > def f(): > > exec "y=2" > > def g(): > > return y > > return g() > > > > print f() > > </frag> > > > > # print 3. > > > > Is that confusing for users? maybe they will more naturally expect 2 > > as outcome (given nested scopes). > > This seems the best compromise to me. It will lead to the least > broken code, because this is the behavior that we had before nested > scopes! It is also quite easy to implement given the current > implementation, I believe. > > Maybe we could introduce a warning rather than an error for this > situation though, because even if this behavior is clearly documented, > it will still be confusing to some, so it is better if we outlaw it in > some future version. > Yes this can be easy to implement but more confusing situations can arise: <frag> y=3 def f(): y=9 exec "y=2" def g(): return y return y,g() print f() </frag> What should this print? the situation leads not to a canonical solution as class def scopes. or <frag> def f(): from foo import * def g(): return y return g() print f() </frag> [Mark Hammond] > > This probably won't be a very popular suggestion, but how about pulling > > nested scopes (I assume they are at the root of the problem) > > until this can be solved cleanly? > > Agreed. While I think nested scopes are kinda cool, I have lived without > them, and really without missing them, for years. At the moment the cure > appears worse then the symptoms in at least a few cases. If nothing else, > it compromises the elegant simplicity of Python that drew me here in the > first place! > > Assuming that people really _do_ want this feature, IMO the bar should be > raised so there are _zero_ backward compatibility issues. I don't say anything about pulling nested scopes (I don't think my opinion can change things in this respect) but I should insist that without explicit syntax IMO raising the bar has a too high impl cost (both performance and complexity) or creates confusion. [Andrew Kuchling] > >Assuming that people really _do_ want this feature, IMO the bar should be > >raised so there are _zero_ backward compatibility issues. > > Even at the cost of additional implementation complexity? At the cost > of having to learn "scopes are nested, unless you do these two things > in which case they're not"? > > Let's not waffle. If nested scopes are worth doing, they're worth > breaking code. Either leave exec and from..import illegal, or back > out nested scopes, or think of some better solution, but let's not > introduce complicated backward compatibility hacks. IMO breaking code would be ok if we issue warnings today and implement nested scopes issuing errors tomorrow. But this is simply a statement about principles and raised impression. IMO import * in an inner scope should end up being an error, not sure about 'exec's. We will need a final BDFL statement. regards, Samuele Pedroni.

18 78

deprecating string module?
by Skip Montanaro 12 Jul '02

12 Jul '02

I know this has been discussed before. In light of the recent PEP8 update: Avoid the use of the string module; instead use string methods. These are always much faster and share the same API with unicode strings. I thought I'd raise it again. How about deprecating the string module? In the past, this has presented two major problems. First, its near ubiquity in Python programs before the availability of string methods. Second, the presence of a few useful data objects (digits, uppercase, etc). The first problem can be solved by extending the deprecation time suitably (two years/four releases?) I think the second problem can be solved by adding those data objects to either sys or locale(preferably the latter). string.letters and its subsets are locale-dependent. In relevant locales I presume string.punctuation and string.whitespace would also be different than their common ASCII elements. Finally, I'd amend the above PEP8 admontion something like Avoid the use of the string module unless backward-compatibility with versions earlier than Python 2.0 is important; instead use string methods. These are always much faster and share the same API with unicode strings. Skip

22 118

Re: [Python-Dev] trace.py and the obscurity of Tools/scripts/ (was: Unittest list)
by Michael Hudson 06 Jun '02

06 Jun '02

Zooko <zooko(a)zooko.com> writes: > Since even python-dev'ers find ancient copies of trace.py on ftp > sites before they find the one in the Tools/scripts/ directory, and > since the debian package of Python 2.2 comes without a > Tools/scripts/ directory at all, I conclude that the Tools/scripts/ > directory it isn't doing its job very well. I suggest it either be > killed or fixed. I'm not sure how to do the latter -- link to it > from the doc pages? Kill it and I'll be after you with an axe. Admittedly there's a lot of cruft that should be removed, rewritten or reorganised, but there's some mighty useful stuff there too (eg. logmerge.py). I don't see why I should suffer just because other people don't know where to look... Cheers, M. -- Darned confusing, unless you have that magic ingredient coffee, of which I can pay you Tuesday for a couple pounds of extra-special grind today. -- John Mitchell, 11 Jan 1999

9 16

Socket timeout patch
by Michael Gilfix 06 Jun '02

06 Jun '02

The socket timeout patch is finally available with doc updates n' unit test as patch #555085 in the SF tracker. This implementation provides timeout functionality at the C level. A patch to socket.py also fixes the problem of losing data while an exception is thrown to the underlying socket. -- Mike -- Michael Gilfix mgilfix(a)eecs.tufts.edu For my gpg public key: http://www.eecs.tufts.edu/~mgilfix/contact.html

3 10

Python 2.3 release schedule
by Guido van Rossum 04 Jun '02

04 Jun '02

The Python 2.3 release schedule as laid out in PEP 283 is unrealistic. It has May 22 (yesterday) as the release date for 2.3a1, and then promises successive releases at approximately 4-week intervals. I think we're nowhere near doing the first alpha release. PythonLabs has been consumed by high priority Zope projects. There's a little less pressure now, and we can go back to thinking about Python releases, but we've lost a lot of time. We haven't even appointed a release manager! (Or the appointment was never recorded in the PEP.) In the discussion about release stability, it was brought up that the release schedule should be driven by feature completeness, not by the calendar. I basically agree, although I believe that when it takes the responsible party for a particular feature forever to implement it, it's better to move the feature to future release than to hold up the release forever. So let's discuss features for 2.3. PEP 283 mentions these: > PEP 266 Optimizing Global Variable/Attribute Access Montanaro > PEP 267 Optimized Access to Module Namespaces Hylton > PEP 280 Optimizing access to globals van Rossum (These are basically three friendly competing proposals.) Jeremy has made a little progress with a new compiler, but it's going slow and the compiler is only the first step. Maybe we'll be able to refactor the compiler in this release. I'm tempted to say we won't hold our breath for this one. > PEP 269 Pgen Module for Python Riehl I haven't heard from Jon Riehl, so I consider dropping this idea. > PEP 273 Import Modules from Zip Archives Ahlstrom I think this is close -- maybe it's already checked in and I don't know about it! > PEP 282 A Logging System Mick Vinay Sajip has been making steady progress on an implementation, and despite a recent near-flamewar (which I haven't followed) I expect that his code will be incorporated in the near future. That's it as far as what PEP 283 mentions (there's also a bunch of things already implemented, like bool, PyMalloc, universal newlines). Here are some ideas of my own for things I expect to see in 2.3a1: - Provide alternatives for common uses of the types module; essentially Skip's proto-PEP. - Extended slice notation for all built-in sequences. Wasn't Raymond Hettinger working on this? - Fix the buffer object??? http://mail.python.org/pipermail/python-dev/2002-May/023896.html - Lazily tracking tuples? Neil, how's that coming? http://mail.python.org/pipermail/python-dev/2002-May/023926.html - Timeoutsocket. Work in progress. http://mail.python.org/pipermail/python-dev/2002-May/024077.html - Making None a keyword. Can't be done right away, but a warning would be a first step. http://mail.python.org/pipermail/python-dev/2002-April/023600.html - Stage 2 of the int/long integration (PEP 237). This mostly means warning about situations where hex, oct or shift of an int returns a different value than for the same value as a long. (I think the PEP misses this step, but it's necessary -- we can't just change the semantics silently without warning first.) - I think Andrew Kuchling has a bunch of distutils features planned; how's that coming? - PEP 286??? (MvL: Enhanced argument tuples.) I haven't had the time to review this thoroughly. It seems a deep optimization hack (also makes better correctness guarantees though). - A standard datetime type. An implementation effort is under way: http://www.zope.org/Members/fdrake/DateTimeWiki/FrontPage Effbot and MAL have a proposal for a basic interface that all datetime types should implement, but there are some problems with UTC. A decision needs to be made. Anything else? --Guido van Rossum (home page: http://www.python.org/~guido/)

10 22

Adding Optik to the standard library
by Greg Ward 04 Jun '02

04 Jun '02

I think it's time to declare the work of the getopt-sig finished: several competing proposals were put forward, but Optik appears to be the only really complete, documented, field-tested (by someone other than its author) library. Not everyone on the getopt-sig will agree, but I think that's the broad consensus. Take this with a grain of salt, though -- I'm biased. ;-) Anyways, I think further consensus is needed on how precisely to add Optik to the standard library. The only constraint I've heard from Guido is to give it a less-cutesy name, which is fine by me. First, background: Optik consists of three modules: optik.option, optik.option_parser, and optik.errors, but that detail is hidden from users -- Optik applications normally just do this: from optik import OptionParser although there are a handful of other names that are occasionally useful to import from the 'optik' package: Option, SUPPRESS_HELP, OptionValueError, etc. Optik's __init__.py file is here: http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/*checkout*/optik/optik/lib/_… It's only about 1100 lines, including docs and comments and blanks, so they could easily be merged into one module if people think that's preferable. So the main issues are 1) where those names should be imported from (interface), and 2) how the standard library should be rearranged to make this interface unobtrusive and efficient (implementation). I'm going to toss out ideas at random until I get bored. Please provide feedback and/or extra ideas on getopt-sig(a)python.org. IDEA #1: interface: from getopt import OptionParser # and the other Optik names implementation: * turn the getopt module into a package * put the current getopt.py into, say, getopt/classic_getopt.py * make getopt/__init__.py import everything from classic_getopt.py and Optik's three modules, so that either interface is there for the asking pros: * dead simple cons: * applications using just the classic getopt interface suddenly find themselves importing lots more code than they used to IDEA #2: interface: from getopt.option_parser import OptionParser, ... implementation: * as before, getopt.py becomes getopt/classic_getopt.py * getopt/__init__.py consists solely of "from classic_getopt import *" * Optik's three modules are copied into getopt, with the right imports added to getopt/option_parser.py so that applications don't have to worry about where Optik's other names come from pros: * only slightly more burden on apps now using classic getopt cons: * interface is a tad clunkier IDEA #3: interface: from getopt.option_parser import OptionParser, SUPPRESS_HELP, ... from getopt.option import Option from getopt.errors import OptionValueError implementation: * classic getopt handled the same as #2 * just dump Optik's three modules into getopt/ and be done with it pros: * dead simple cons: * clunky interface -- imports expose a lot of implementation detail IDEA #4: interface: same as #1 implementation: * same as #1, except use some funky import-time magic to ensure that the Optik code is only imported if someone actually needs it. Barry Warsaw provided a patch to do this: http://sourceforge.net/tracker/index.php?func=detail&aid=544175&group_id=38… pros: * more efficient for apps using classic getopt cons: * more complicated; apparently Guido expressed distaste for the import-time magic. I'm a little leery of it myself, although I have not carefully read the code. Preferences? Other ideas? Surely the right solution is out there somewhere, just beyond my reach... Greg -- Greg Ward - Unix geek gward(a)python.net http://starship.python.net/~gward/ "Passionate hatred can give meaning and purpose to an empty life." -- Eric Hoffer

12 36

Lazily GC tracking tuples
by Daniel Dunbar 04 Jun '02

04 Jun '02

A few days ago the idea of untracking, or somehow removing, immutable and provably uncyclic tuples from garbage collection was bounced about - at the time the idea was to untrack a tuple once it was noticed that it contained no cyclic objects... this proves annoying to implement because of unfilled tuples, and C-api calls that mutate the tuple (highly unusual it seems). Lazily tracking the tuples seems to be a much simpler proposition: a tuple can only be a member of a cycle once an element of ob_item is set to an object that could contain a cycle, presumably items are set in a tuple far less frequently than they are accessed (by normal code or the collector), so I imagine this method is also more efficient. The changes required are fairly small, PyObject_GC_IS_TRACKED is added to determine if an object is tracked, the gc track and untrack operations in tupleobject.c are modified to account for the tuple not always being tracked, and PyTuple_SET_ITEM and PyTuple_SetItem are modified to begin tracking the tuple if a) it is not already tracked, and b) the item being set _could_ contain a cycle. At the moment, I use PyObject_IS_GC as a conserative approximation to '_could_ contain a cycle', which works well except for nested tuples. In theory if v is a tuple, _SetItem(o,i,v) should only be called when v has been completely filled, so if v is untracked then we know it also cannot be member of a cycle, which solves the nested tuple problem. In practice making the nested tuple change did not cause any change in behavior on the test suite, but I have left it out for simplicity. Large trees of atomic data represented as tuples seems like a case where it could possibly make a difference. A 2.23 patch for the changes outlined above are at: http://www.geocities.com/evilzr/lazytuplegc/ (I wasn't sure if such things are appropriate for the SF-patch manager - yes/no?) If you want to actually test the changes you need a recent checkout, there was a minor collector bug that the lazy tracking exploited heavily. Some stats - cvs | lazytuplegc --------------------------------------- iterzip_test, N=500000 juststore: 1.46 | 1.43 gc size 1143 | 788 justtups: 1.40 | 1.30 gc size 1143 | 788 storetups: 8.48 | 4.11 gc size 501143 | 788 base gc set size python: 1081 | 742 idle: 7754 | 3818 pybench 1.0 avg time (n=10,warp=20) 69813.40 | 69708.04 The iterzip_test is only slightly munged from the one Tim Peters posted on the iterzip() thread. The pybench averages are almost identical, however there are large fluctuations on the individual tests, see http://www.geocities.com/evilzr/lazytuplegc/cmp.txt The base gc set numbers are obtained by starting python (or idle) and immediatly running a collection with gc.DEBUG_STATS on. All gc size numbers are size(gen0)+size(gen1)+size(gen2). this patch causes one regrtest to fail (test_gc, in test_staticmethod, for reasons I do not yet understand (help!)). ... make of it what you will :) Oh and, not to make this post longer, but heres the first-pydev-list-posting-info: I'm Daniel Dunbar, my introduction to Python was embedding it into the Blender 3D package (www.blender3d.com, but we went out of business - oops). Since then I have used it for lots of esoteric little tasks, but only recently after wrapping some of my C-based libraries have I started writing complex (ie. applications not utilities) programs with it. My main interest related to python development is making it go faster (so I can optimize my code less)... but thats probably not rare. ===== daniel dunbar evilzr(a)yahoo.com __________________________________________________ Do You Yahoo!? Yahoo! Health - your guide to health and wellness http://health.yahoo.com

6 19

Other library code transformations
by Raymond Hettinger 03 Jun '02

03 Jun '02

While we're eliminating uses of the string and types modules, how about other code clean-ups and modernization: d.has_key(k) --> k in d if k in d.keys() --> if k in d obj.__dict__ --> vars(obj) class X(UserList) --> class X(list) class X(UserDict) --> class X(dict) # and remove .data references 0L --> 0 1L --> 1 lambda x, y=y: fun(x,y) --> lambda x: fun(x,y) x = y.__class__ --> x is type(y) == None --> is None if item in astring --> if item in adict This is from my own list of code updates, the python library already in better shape. Raymond Hettinger

8 23

Re: [Python-Dev] Python 2.3 release schedule
by Michael Hudson 01 Jun '02

01 Jun '02

Tim Peters <tim.one(a)comcast.net> writes: > [Michael Hudson] > > Surprise of the day: this [is all that's required!]: > > > > #if PY_VERSION_HEX < 0x01060000 > > /* raw memory interface already present */ > > > > /* there is no object memory interface in 1.5.2 */ > > #define PyObject_Malloc(size) PyMem_Malloc((size)) > > #define PyObject_Realloc(p, size) PyMem_Realloc((p), (size)) > > ... > > How come not the simpler > > #define PyObject_Malloc PyMem_Malloc > #define PyObject_Realloc PyMem_Realloc > > etc instead? Filling in an argument list prevents use of a name as a > function designator. I thought filling out the arguments was better style, for some reason. Easy enough to change. > > ... > > /* There are three "families" of memory API: the "raw memory", "object > > memory" and "object" families. > > > > Raw Memory: > > > > PyMem_Malloc, PyMem_Realloc, PyMem_Free > > > > Object Memory: > > > > PyObject_Malloc, PyObject_Realloc, PyObject_Free > > > > Object: > > > > PyObject_New, PyObject_NewVar, PyObject_Del > > I rarely mention the PyObject_GC_XYZ family because it's already in good > shape. But its API changed between 2.1 and 2.2, and it would be good to > supply a compatibility layer for it too. Pick Neil's brain <wink>. I was under the impression that the 2.1 and 2.2 interfaces differed in ways that couldn't easily be papered over with macros. I'll check. > > ... > > Comments, flames, etc appreciated. > > I especially liked the (snipped) to-the-point comment blocks; Thanks; I think it is a good idea to describe intended usage in no uncertain terms *somewhere* at least. Probably lots of places. Any book authors reading python-dev? > PyObject_Del problems were already mentioned; thank you! I hope it helps! Cheers, M. -- Famous remarks are very seldom quoted correctly. -- Simeon Strunsky

3 2

Customization docs
by David Abrahams 01 Jun '02

01 Jun '02

A few questions about customization and the docs, TIA: ------ http://www.python.org/dev/doc/devel/ref/customization.html tosses off this sentence in the section on rich comparisons: "A rich comparison method may return NotImplemented if it does not implement the operation for a given pair of arguments." ...but it says nothing about the effect that will have. Using Thomas Heller's search engine, I found http://www.python.org/dev/doc/devel/ref/types.html#l2h-59, which says "...Numeric methods and rich comparison methods may return this value if they do not implement the operation for the operands provided. (The interpreter will then try the reflected operation, or some other fallback, depending on the operator.) " This seems like the wrong place to bury the description of the mechanism for operating on heterogeneous arguments. Shouldn't a short description go into the Customization section of the docs? ------- It also mentions, "There are no reflected (swapped-argument) versions of these methods (to be used when the left argument does not support the operation but the right argument does)..." since these are the first binary operators described the whole concept of reflected methods hasn't been introduced yet, and the user has no reason to think that reflected versions should exist. [incidentally, "reflected" and "reflection" produce no hits with Thomas' engine] -------

3 6