Mailman 3 April 2007 - Python-ideas

Ideas towards GIL removal
by Greg Ewing July 18, 2008

July 18, 2008

I've been thinking about some ideas for reducing the amount of refcount adjustment that needs to be done, with a view to making GIL removal easier. 1) Permanent objects In a typical Python program there are many objects that are created at the beginning and exist for the life of the program -- classes, functions, literals, etc. Refcounting these is a waste of effort, since they're never going to go away. So perhaps there could be a way of marking such objects as "permanent" or "immortal". … [View More]

8 19

PEP for executing a module in a package containing relative imports
by Brett Cannon May 7, 2007

May 7, 2007

Some of you might remember a discussion that took place on this list about not being able to execute a script contained in a package that used relative imports (read the PEP if you don't quite get what I am talking about). The PEP below proposes a solution (along with a counter-solution). Let me know what you think. I especially want to hear which proposal people prefer; the one in the PEP or the one in the Open Issues section. Plus I wouldn't mind suggestions on a title for this PEP. =) --… [View More]----------------------------------------- PEP: XXX Title: XXX Version: $Revision: 52916 $ Last-Modified: $Date: 2006-12-04 11:59:42 -0800 (Mon, 04 Dec 2006) $ Author: Brett Cannon Status: Draft Type: Standards Track Content-Type: text/x-rst Created: XXX-Apr-2007 Abstract ======== Because of how name resolution works for relative imports in a world where PEP 328 is implemented, the ability to execute modules within a package ceases being possible. This failing stems from the fact that the module being executed as the "main" module replaces its ``__name__`` attribute with ``"__main__"`` instead of leaving it as the actual, absolute name of the module. This breaks import's ability to resolve relative imports from the main module into absolute names. In order to resolve this issue, this PEP proposes to change how a module is delineated as the module that is being executed as the main module. By leaving the ``__name__`` attribute in a module alone and setting a module attribute named ``__main__`` to a true value for the main module (and thus false in all others), proper relative name resolution can occur while still having a clear way for a module to know if it is being executed as the main module. The Problem =========== With the introduction of PEP 328, relative imports became dependent on the ``__name__`` attribute of the module performing the import. This is because the use of dots in a relative import are used to strip away parts of the calling module's name to calcuate where in the package hierarchy a relative import should fall (prior to PEP 328 relative imports could fail and would fall back on absolute imports which had a chance of succeeding). For instance, consider the import ``from .. import spam`` made from the ``bacon.ham.beans`` module (``bacon.ham.beans`` is not a package itself, i.e., does not define ``__path__``). Name resolution of the relative import takes the caller's name (``bacon.ham.beans``), splits on dots, and then slices off the last n parts based on the level (which is 2). In this example both ``ham`` and ``beans`` are dropped and ``spam`` is joined with what is left (``bacon``). This leads to the proper import of the module ``bacon.spam``. This reliance on the ``__name__`` attribute of a module when handling realtive imports becomes an issue with executing a script within a package. Because the executing script is set to ``'__main__'``, import cannot resolve any relative imports. This leads to an ``ImportError`` if you try to execute a script in a package that uses any relative import. For example, assume we have a package named ``bacon`` with an ``__init__.py`` file containing:: from . import spam Also create a module named ``spam`` within the ``bacon`` package (it can be an empty file). Now if you try to execute the ``bacon`` package (either through ``python bacon/__init__.py`` or ``python -m bacon``) you will get an ``ImportError`` about trying to do a relative import from within a non-package. Obviously the import is valid, but because of the setting of ``__name__`` to ``'__main__'`` import thinks that ``bacon/__init__.py`` is not in a package since no dots exist in ``__name__``. To see how the algorithm works, see ``importlib.Import._resolve_name()`` in the sandbox [#importlib]_. Currently a work-around is to remove all relative imports in the module being executed and make them absolute. This is unfortunate, though, as one should not be required to use a specific type of resource in order to make a module in a package be able to be executed. The Solution ============ The solution to the problem is to not change the value of ``__name__`` in modules. But there still needs to be a way to let executing code know it is being executed as a script. This is handled with a new module attribute named ``__main__``. When a module is being executed as a script, ``__main__`` will be set to a true value. For all other modules, ``__main__`` will be set to a false value. This changes the current idiom of:: if __name__ == '__main__': ... to:: if __main__: ... The current idiom is not as obvious and could cause confusion for new programmers. The proposed idiom, though, does not require explaining why ``__name__`` is set as it is. With the proposed solution the convenience of finding out what module is being executed by examining ``sys.modules['__main__']`` is lost. To make up for this, the ``sys`` module will gain the ``main`` attribute. It will contain a string of the name of the module that is considered the executing module. A competing solution is discussed in `Open Issues`_. Transition Plan =============== Using this solution will not work directly in Python 2.6. Code is dependent upon the semantics of having ``__name__`` set to ``'__main__'``. There is also the issue of pre-existing global variables in a module named ``__main__``. To deal with these issues, a two-step solution is needed. First, a Py3K deprecation warning will be raised during AST generation when a global variable named ``__main__`` is defined. This will help with the detection of code that would reset the value of ``__main__`` for a module. Without adding a warning when a global variable is injected into a module, though, it is not fool-proof. But this solution should cover the vast majority of variable rebinding problems. Second, 2to3 [#2to3]_ will gain a rule to transform the current ``if __name__ == '__main__': ...`` idiom to the new one. While it will not help with code that checks ``__name__`` outside of the idiom, that specific line of code makes up a large proporation of code that every looks for ``__name__`` set to ``'__main__'``. Open Issues =========== A counter-proposal to introducing the ``__main__`` attribute on modules was to introduce a built-in with the same name. The value of the built-in would be the name of the module being executed (just like the proposed ``sys.main``). This would lead to a new idiom of:: if __name__ == __main__: ... The perk of this idiom over the one proposed earlier is that the general semantics does not differ greatly from the current idiom. The drawback is that the syntactic difference is subtle; the dropping of quotes around "__main__". Some believe that for existing Python programmers bugs will be introduced where the quotation marks will be put on by accident. But one could argue that the bug would be discovered quickly through testing as it is a very shallow bug. The other pro of this proposal over the earlier one is the alleviation of requiring import code to have to set the value of ``__main__``. By making it a built-in variable import does not have to care about ``__main__`` as executing the code itself will pick up the built-in ``__main__`` itself. This simplies the implementation of the proposal as it only requires setting a built-in instead of changing import to set an attribute on every module that has exactly one module have a different value (much like the current implementation has to do to set ``__name__`` in one module to ``'__main__'``). References ========== .. [#2to3] 2to3 tool (http://svn.python.org/view/sandbox/trunk/2to3/) [ViewVC] .. [#importlib] importlib (http://svn.python.org/view/sandbox/trunk/import_in_py/importlib.py?view=mar…) [ViewVC] Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: [View Less]

14 58

fixing mutable default argument values
by Chris Rebert May 3, 2007

May 3, 2007

If A.M. Kuchling's list of Python Warts is any indication, Python has removed many of the warts it once had. However, the behavior of mutable default argument values is still a frequent stumbling-block for newbies. It is also present on at least 3 different lists of Python's deficiencies ([0][1][2]). Example of current, unintuitive behavior (snipped from [0]): >>> def popo(x=[]): ... x.append(666) ... print x ... >>> popo() [666] >>> popo() [666, 666] … [View More]

9 25

ordered dict
by Mathias Panzenböck May 1, 2007

May 1, 2007

Some kind of ordered dictionary would be nice to have in the standard library. e.g. a AVL tree or something like that. It would be nice so we can do things like that: for value in tree[:end_key]: do_something_with(value) del tree[:end_key] A alternative would be just to sort the keys of a dict but that's O( n log n ) for each sort. Depending on what's the more often occurring case (lookup, insert, get key-range, etc.) a other kind of dict object would make sense. What do you think? -panzi

5 18

Python package files
by Adam Atlas April 30, 2007

April 30, 2007

I think it would be useful for Python to accept imports of standalone files representing entire packages, maybe with the extension .pyp. A package file would basically be a ZIP file, so it would follow fairly easily from the current zipimport mechanism... its top-level directory would be the contents of a package named by the outer ZIP file. In other words, suppose we have a ZIP file called "package.pyp", and at its top level, it contains "__init__.py" and "blah.py". Anywhere this … [View More]

5 5

Re: [Python-ideas] Minor suggestion for unittest
by Terry Jones April 29, 2007

April 29, 2007

Hi Collin Thanks for the reply. | It sounds like what you're looking for is FunctionTestCase | (http://docs.python.org/lib/unittest-contents.html). Using that, your | loop above becomes something like | | for testFunc, expectedResult in MyTestData: | def tester(): | self.assertEqual(testFunc(), expectedResult) | suite.addTest(FunctionTestCase(tester)) I had read about FunctionTestCase but it didn't seem to be what I was looking for - though it's the closest. … [View More]

2 1

The case against static type checking, in detail (long)
by Talin April 27, 2007

April 27, 2007

(This is a fragment of an email that I sent to Guido earlier, I mention this here so that Guido can skip reading it. Of course, I recognize that most people here already know all this - but since it relates to the recent discussion about the value of type checking, I'd like to post it here as a kind of "manifesto" of why Python is the way it is.) Strongly-typed languages such as C++ and Java require up-front declaration of everything. It is the nature of such languages that there is a … [View More]lot of cross-checking in the compiler between the declaration of a thing and its use. The idea is to prevent programmer errors by insuring internal consistency. However, we find in practice that much of the programmer's effort is spent in maintaining this cross-checking structure. To use a building analogy, a statically-typed program is like a truss structure, where there's a complex web of cross-braces, and a force applied at any given point is spread out over the whole structure. Each time such a program is modified, the programmer must partially dismantle and then re-assemble the existing structure. This takes time. It also clutters the code. Reading the source code of a program written in a statically typed language reveals that a substantial part of the text serves only to support the compile-time checking of definitions, or provides visual redundancy to aid the programmer in connecting two aspects of the program which are defined far apart. An example of what I mean is the use of variable type declarations - even in a statically typed language, it would be fairly easy for the compiler to automatically infer most variable types if the language were designed that way; The fact that the programmer is required to manually specify these types serves as an additional consistency check on the code. However, time spend serving the needs of these consistency checks is time away from actually serving the functional purpose of the code. Programmers in Python, on the other hand, not only need not worry about type declarations, they also spend much less time worrying about converting from one type to another in order to meet the constraints of a particular API. This is one of the reasons why I can generally write Python code about 4 times as fast as the C++ equivalent. (Understand that this is coming from someone who loves working in C++ and Java and has used them daily for the last 15 years. At the same time, however, I also enjoy programming in Python and I recognize that each language has their strengths.) There is also the question of how much static typing helps improve program reliability. In statically typed languages, there are two kinds of ways that types are used. In languages such as C and Pascal, the type declarations serve primarily as a consistency check. However, in C++ template metaprogramming, and in languages like Haskell, there is a second use for types, which is to provide a kind of type calculus or type inferencing, which gives additional expressive power to the language. C++ templates can act as powerful code generators, allowing the programmer to program in ever higher levels of abstraction and express the basic ideas even more succinctly and clearly than before. In a rapid-prototyping environment, the second use of types can be a major productivity win; However I would argue that the first use of types, consistency checking, is less beneficial, and is often more of a distraction to the programmer than a help. Yes, static type checking does detect some errors; But it also causes errors by making the code larger and more wordy, because that the programmer cannot hold large portions of the program in their mind all at once, which can lead to errors in overall design. It means the programmer spends more time thinking about the behavior of individual variables and less about the algorithm as a whole. At this point, I want to talk about a related matter, another fundamental design aspect of Python which I call "decriminalization of minor errors". An example of this is illustrated by the recent discussion over string slicing. As you know, when you attempt to index a string with a slice object that extends outside of the bounds of the string, the range is silently truncated. Some argued that Python should be more strict, and report an error when this occurs - but instead, it was reaffirmed that the current behavior is correct. I would agree that this current behavior is the more Pythonic, and is part of a general pattern, which I shall attempt to describe: To "decriminalize" an error means to find some alternative interpretation of the programmer's intent that produces a non-error result. That is, given a syntactical construct, and a choice of several interpretations of what that construct should mean, attempt to pick an interpretation that, when executed, does not produce an error. In the design of the Python language, it is a regular practice to decriminalize minor errors, provided that the alternative interpretation can meet some fairly strict criteria: That it is useful, intuitive, reasonable, and pedagogically sound. Note that this is a much more conservative rule than that used by languages such as Rexx, Javascript, and Perl, languages which make "heroic efforts" to bend the interpretation of an operation to a non-error result. Python does not do this. Nor is decriminalizing errors isn't the same as ignoring errors. Errors are still, and should be, enforced vigorously and reported. The distinction is that decriminalizing an error results in code that produces a useful, logical result, whereas ignoring errors results in code that produces garbage or nothing. Decriminalization comes about when we broaden our definitions of what is the correct result of a given operation. A couple of other examples of decriminalization: 1) there are languages in which the only valid argument for an 'if' statement is a boolean. Attempts to say "if 0" are errors. In Python we relax that rule, allowing any type to be used as the argument to an if-statement. We do this by having a broader interpretation of what it means to test an object for 'trueness', and allow 'trueness' to be implied by 'non-emptiness'. 2) Duck-typing is a decriminalization of the error that polymorphic types are required to inherit from a common interface. It also decriminalizes "missing methods", as long as those methods are never called. Again, this is due to having a broader interpretation of 'polymorphism'. (In fact, this aspect of Python is so fundamental, that I think that it deserves its own acronym alongside TOOWTDI and others, but I can't think of a short, pithy description of it. Maybe IOANEIR - "Interpret operations as non-errors if reasonable.") Both static typing and decriminalization serve the same end - telling the programmer "don't sweat the small stuff". Both are very helpful and powerful, because they allow programmers to spend much less time worrying about minor error cases, things that would have to be checked for in C++. Python code is simply more *concise* than the C++ equivalent, yet it achieves this without being terse and cryptic, because the text of a Python program more closely embodies the "essence" of an algorithm, uncluttered by other agendas. The price we pay for this, of course, is that sometimes errors show up much later (like, after ship) than they would have otherwise. But unit testing can catch a lot of the same errors. And in many cases, the seriousness of such errors depends on what we mean by "ship". It's one thing to discover a fatal error after you've pressed thousands of CDs and shipped them all over the world; It's a much different matter if the program has the ability to automatically update itself, or is downloaded from some kind of subscription model such as a package manager. In many environments, it is far more important to get something done quickly and validate the general concept, than it is to insure that the code is 100% correct. In other words, if it would take you 6 months to write it in a statically typed language, but only 2 months to write it in a dynamic language - well, that's 4 extra months you have to write unit tests and make sure it's right! And in the mean time, you can have real users banging on the code and making sure of something that is far more important, which is whether what you wrote is the right thing at all. -- Talin [View Less]

6 5

What would you call such an object (was: ordered dict)
by LD 'Gus' Landis April 26, 2007

April 26, 2007

Hi, I am wondering if in y'alls opinion the following is "just a dictionary" or is it a different kind of object that has some of the characteristics of a dictionary, and has order. If y'all think that it is "just a dictionary", then how does one override the notion of a "hash" for the key in a dictionary and make it some other ordered structure (e.g. a B-Tree, AVL, etc). (Please no flame toss to some other list -- this is a "use" of an ordered "ordered dict") I don't know … [View More]what such a critter would be called (in Python). It has the name of "array" language where it is central, but don't want to go into that. The object has the following characteristics: - It is indexed by keys which are immutable (like dicts) - Each key has a single value (like dicts) - The keys are ordered (usually a B-Tree underneath) - The keys are "sorted" yielding a hierarchy such that (using Python tuples as an example and pseudo Python): object = { (0,): "value of node", (0,"name") : "name of node", (0,"name",1): "some data about name", (1,): "value of another node", (1,2,3): "some data value", (2,): 2, (2,2,"somekey",1): 32, (3,): 28, ("abc",1,2): 14 } - Introspection of the object allows walking the keys by hierarchy, using the above: key = object.order(None) -> 0 key = object.order(key) -> 1 key = object.order(key) -> 2 key = object.order(key) -> 3 key = object.order(key) -> "abc" key = object.order(key) -> None The first key is fetched when (None) is the initial key (or last key if modifier is -1) Supplying a modifier (-1, where 1 is default of forward, -1 is reverse) in the call traverses the keys in the reverse order from that shown above. - Introspection of the key results in: hasdata = object.data(key) =0 no subkeys no data for 'key' (in the above (39) would have no subkeys, no data) =1 no subkeys has data for 'key' (in the above (3) has no subkeys, but has data) =10 has subkeys no data for 'key' (in the above (2,2) has subkeys but no data) =11 has subkeys has data for 'key' (in the above (2) has subkeys and has data) - Introspection of object can yield "depth first" keys key = object.query(None) -> (0,) key = object.query(key) -> (0,"name") key = object.query(key) -> (0,"name",1) key = object.query(key) -> (1,) key = object.query(key) -> (1,2,3) key = object.query(key) -> (2,) key = object.query(key) -> (2,2,"somekey",1) key = object.query(key) -> (3,) key = object.query(key) -> ("abc",1,2) key = object.query(key) -> None Like object.order(), object.query() has the same "reverse" (using -1) option to walk the keys in a reverse order. - Having an iterator over order/query: for key in object.ordered([start[,end]): for key in object.queryed([start[,end]): (spelling?? other alternative) - Set/get of object[(0,"name")] = "new name of node" print object[(0,"name")] Cheers, --ldl -- LD Landis - N0YRQ - de la tierra del encanto 3960 Schooner Loop, Las Cruces, NM 88012 651/340-4007 N32 21'48.28" W106 46'5.80" "If a thing is worth doing, it is worth doing badly." –GK Chesterton. An interpretation: For things worth doing: Doing them, even if badly, is better than doing nothing perfectly (on them). [View Less]

2 1

Minor suggestion for unittest
by Terry Jones April 26, 2007

April 26, 2007

There's a simple change that could be made to unittest that would make it easier to automate some forms of testing. I want to be able to dynamically add tests to an instance of a class derived from unittest.TestCase. There are occasions when I don't want to write my tests upfront in a Python file. E.g., given a bunch of test/expectedResult data sitting around (below in a variable named MyTestData), it would be nice to be able to do the following (untested here, but I did it earlier for real … [View More]and it works fine): import unittest class Test(unittest.TestCase): def runTest(): pass suite = unittest.TestSuite() for testFunc, expectedResult in MyTestData: newTestFuncName = 'dynamic-test-' + testFunc.__name__ def tester(): self.assertEqual(testFunc(), expectedResult) test = Test() setattr(test, newTestFuncName, tester) # Set the class instance up so that it will be the one run. test.__init__(newTestFuncName) # ugh! suite.addTest(test) suite.run() The explicit call to __init__ (marked ugh!) is ugly, dangerous, etc. You could also say test._testMethodName = newTestFuncName (and set _testMethodDoc too), but that's also ugly. This would all be very simple though if instead of starting out like: class TestCase: def __init__(self, methodName='runTest'): try: self._testMethodName = methodName testMethod = getattr(self, methodName) self._testMethodDoc = testMethod.__doc__ except AttributeError: raise ValueError, "no such test method in %s: %s" % \ (self.__class__, methodName) unittest.TestCase started out like this: class TestCase: def __init__(self, methodName='runTest'): self.setTestMethod(methodName) def setTestMethod(self, methodName): try: self._testMethodName = methodName testMethod = getattr(self, methodName) self._testMethodDoc = testMethod.__doc__ except AttributeError: raise ValueError, "no such test method in %s: %s" % \ (self.__class__, methodName) That would allow people to create an instance of their Test class, add a method to it using setattr, and then use setTestMethod to set the method to be run. A further improvement would be to have _testMethodName be None or left undefined (and accessed via __getattr__) for as long as possible rather than being set to runTest (and looked up with getattr) immediately. That would allow the removal of the do-nothing runTest method in the above. No old code need be broken as runTest would still be the default. You'd just have a chance to get in there earlier so it never saw the light of day. Programmers like to automate things, especially testing. These changes don't break any existing code but they allow additional test automation. Of course you _could_ achieve the above by writing out a brand new temp.py file, running it, and so on, but that's not very Pythonic, is a bunch more work, needs cleanup (temp.py needs to go away), etc. I have some further thoughts about how to make this a bit more flexible, but I'll save those for later, supposing there's any interest in the above. Terry Jones [View Less]

2 1

Sandbox?
by Mathias Panzenböck April 24, 2007

April 24, 2007

Are there any plans on a sandbox for python 3.0? Just wondering. -panzi

2 1

Removing instancemethod in favor of partial?
by Calvin Spealman April 24, 2007

April 24, 2007

Hey, why not? They do basically the same thing, except instancemethod allows only a single argument. Why not allow class and instance methods to be wrapped with a partial instead of their own type? We can rip out 300 lines of C code supporting instance method, at least. The only thorn is the im_class attribute, but few seem to even use it (few meaning just Twisted, according to Google Code Search). Anyway, I figure we don't really need it anyway because if some of the proposals for a way to … [View More]

2 1

partial with skipped arguments
by Calvin Spealman April 23, 2007

April 23, 2007

I often wish you could bind to arguments in a partial out of order, skipping some positionals. The solution I came up with is a singleton object located as an attribute of the partial function itself and used like this: def foo(a, b): return a / b pf = partial(foo, partial.skip, 2) assert pf(1.0) == 0.5 -- Read my blog! I depend on your acceptance of my opinion! I am interesting! http://ironfroggy-code.blogspot.com/

3 5

Object adaptation and interfaces and so forth
by Adam Atlas April 23, 2007

April 23, 2007

(Not exactly an idea post, but I don't want to bother python-dev or python-3000 with this.) PEP 246 was rejected a year or so ago, and Guido's rejection note stated "Something much better is about to happen; it's too early to say exactly what, but it's not going to resemble the proposal in this PEP." Does anyone know if anything has gone on with this concept since then? It seems like it has a lot of really interesting potential, although I do see why PEP 246's specific proposal was … [View More]

3 2

PEP 30xx: Access to Module/Class/Function Currently Being Defined (this)
by Jim Jewett April 23, 2007

April 23, 2007

(Please note that several groups were Cc'd. For now, please limit followups to python-3000. This would *probably* be backported to 2.6, but that wouldn't be decided until the implementation strategy was settled.) PEP: 30XX Title: Access to Module/Class/Function Currently Being Defined (this) Version: $Revision$ Last-Modified: $Date$ Author: Jim J. Jewett <jimjjewett(a)gmail.com> Status: Draft Type: Standards Track Content-Type: text/plain Created: 22-Apr-2007 Python-Version: 3.0 Post-… [View More]History: 22-Apr-2007 Abstract It is common to need a reference to the current module, class, or function, but there is currently no entirely correct way to do this. This PEP proposes adding the keywords __module__, __class__, and __function__. Rationale Many modules export various functions, classes, and other objects, but will perform additional activities (such as running unit tests) when run as a script. The current idiom is to test whether the module's name has been set to magic value. if __name__ == "__main__": ... More complicated introspection requires a module to (attempt to) import itself. If importing the expected name actually produces a different module, there is no good workaround. Proposal: Add a __module__ keyword which refers to the module currently being defined (executed). (But see open issues.) if __module__ is sys.main: ... # assuming PEP 3020, Cannon Class methods are passed the current instance; from this they can determine self.__class__ (or cls, for classmethods). Unfortunately, this reference is to the object's actual class, which may be a subclass of the defining class. The current workaround is to repeat the name of the class, and assume that the name will not be rebound. class C(B): def meth(self): super(C, self).meth() # Hope C is never rebound. class D(C): def meth(self): super(C, self).meth() # ?!? issubclass(D,C), so it "works" Proposal: Add a __class__ keyword which refers to the class currently being defined (executed). (But see open issues.) class C(B): def meth(self): super(__class__, self).meth() Note that super calls may be further simplified by PEP 30XX, Jewett. The __class__ (or __this_class__) attribute came up in attempts to simplify the explanation and/or implementation of that PEP, but was separated out as an independent decision. Note that __class__ (or __this_class__) is not quite the same as the __thisclass__ property on bound super objects. The existing super.__thisclass__ property refers to the class from which the Method Resolution Order search begins. In the above class D, it would refer to (the current reference of name) C. Functions (including methods) often want access to themselves, usually for a private storage location. While there are several workarounds, all have their drawbacks. def counter(_total=[0]): # _total shouldn't really appear in the _total[0] += 1 # signature at all; the list wrapping and return _total[0] # [0] unwrapping obscure the code @annotate(total=0) def counter(): counter.total += 1 # Assume name counter is never rebound return counter.total class _wrap(object): # class exists only to provide storage __total=0 def f(self): self.__total += 1 return self.__total accum=_wrap().f # set module attribute to a bound method Proposal: Add a __function__ keyword which refers to the function (or method) currently being defined (executed). (But see open issues.) @annotate(total=0) def counter(): __function__.total += 1 # Always refers to this function obj return __function__.total Backwards Compatibility While a user could be using these names already, __anything__ names are explicitly reserved to the interpreter. It is therefore acceptable to introduce special meaning to these names within a single feature release. Implementation Ideally, these names would be keywords treated specially by the bytecode compiler. Guido has suggested [1] using a cell variable filled in by the metaclass. Michele Simionato has provided a prototype using bytecode hacks [2]. Open Issues - Are __module__, __class__, and __function__ the right names? In particular, should the names include the word "this", either as __this_module__, __this_class__, and __this_function__, (format discussed on the python-3000 and python-ideas lists) or as __thismodule__, __thisclass__, and __thisfunction__ (inspired by, but conflicting with, current usage of super.__thisclass__). - Are all three keywords needed, or should this enhancement be limited to a subset of the objects? Should methods be treated separately from other functions? References [1] Fixing super anyone? Guido van Rossum http://mail.python.org/pipermail/python-3000/2007-April/006671.html [2] Descriptor/Decorator challenge, Michele Simionato http://groups.google.com/group/comp.lang.python/browse_frm/thread/a6010c749… Copyright This document has been placed in the public domain. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: [View Less]

1 0

Re: [Python-ideas] [Python-Dev] Making builtins more efficient
by Greg Ewing April 20, 2007

April 20, 2007

Steven Elliott wrote: > What I have in mind may be close to what you are suggesting above. My idea is somewhat more uniform and general than that. For the module dict, you use a special mapping type that allows selected items to be accessed by an index as well as a name. The set of such names is determined when the module's code is compiled -- it's simply the names used in that module to refer to globals or builtins. The first time a given builtin is referenced in the module, it will be … [View More]

2 1

Fwd: Implicit String Concatenation
by Tobias Ivarsson April 19, 2007

April 19, 2007

On 4/16/07, Jim Jewett <jimjjewett(a)gmail.com> wrote: > > On 4/13/07, Greg Ewing <greg.ewing(a)canterbury.ac.nz> wrote: > > Josiah Carlson wrote: > > > >>Does anyone have a use case where they *need* > > >>the indentation to be preserved? > > > > Not personally. I think that telling people to > > > use textwrap.dedent() is sufficient. > > > But it seems crazy to make people do this all > > the time, when … [View More]

2 1

Implicit String Concatenation
by Eoghan Murray April 19, 2007

April 19, 2007

I heard the call for the P3K PEP April deadline, so I thought I better get this sent off! When I was first exposed to Python, I was delighted that I could do the following; >>> "Hello" ' world' 'Hello world' This turned to confusion when I tried; >>> domain = " world" >>> "hello" domain Syntax Error ... Invalid Syntax My proposal for Python3K is to allow string-concatenation via juxtaposition between string-literals, string-variables and expressions that … [View More]

15 37

Command line options
by Thomas Heller April 18, 2007

April 18, 2007

Sometimes I think it would be great if it were possible to have standard Python command line options that would allow - initialize and configure the logging module - specify requirements for pkg_resources (for eggs installed with --multi-version All this would avoid having to change logging options or requirements in the script, or having to implement a command line parser for this stuff in every script. The idea is to call python in this way: python --require foo==dev --logging level=… [View More]

2 1

iter() on steroids
by George Sakkis April 14, 2007

April 14, 2007

I proposed an (admittedly more controversial) version of this a few months back at the py3k list and the reaction was unexpectedly (IMO) negative or indifferent, so I'm wondering if things have changed a bit since. The proposal is to make the the builtin iter() return an object with an API that consists of (most) functions currently at itertools. In addition to saving one "from itertools import chain,islice,..." line in every other module I write these days, an extra bonus of the OO interface … [View More]

10 15

Regular expression algorithms
by Adam Atlas April 13, 2007

April 13, 2007

Has anyone seen this article? http://swtch.com/~rsc/regexp/regexp1.html Are its criticisms of Python's regex algorithm accurate? If so, might it be possible to revise Python's `re` module to use this sort of algorithm? I noticed it says that this approach doesn't work if the pattern contains backreferences, but maybe the module could at least sort of self-optimize by switching to this method when no backrefs are used.

3 2

Python and Concurrency
by Talin April 12, 2007

April 12, 2007

I would like to direct your attention to a presentation that was given by Mr. Herb Sutter on the subject of concurrency support in programming languages. It's a long video, but I would urge you to watch at least the first five minutes of it: http://video.google.com/videoplay?docid=7625918717318948700&q=herb+sutter Mr. Sutter is the secretary of the ISO/C++ standards committee. I mention this only so that you'll understand he's not just some random yo-yo with an opinion. (There are a … [View More]number of other papers by Mr. Sutter on the same topic which you can google for, however none of them are as compelling or comprehensive as the video presentation IMHO.) The gist of the introduction is this (my words, not his): Starting from now, and for the rest of your career as a programmer, you better start programming for massively concurrent systems or you are doomed to irrelevance. As he points out in the talk, the good news is that Moore's law is going to continue for several decades to come. The bad news is that most people don't understand Moore's law or what they get from it. Moore's law is really about the number of transistors on a chip. It says nothing about processor speed or performance. He calls it "the end of the free lunch", meaning that in the past you could write a program, wait a little while, and it would run faster - because the hardware got faster while you were waiting. This will no longer be true for single-threaded programs. Scalar processors, with all of their pipelining, prefetching and other tricks, have gotten about as fast as they are ever going to get; We've reached the point of diminishing returns. So the only thing left to do is throw more and more cores on the chip. There's no physical reason why the number of cores won't continue to double every 18 months; the only reason why manufacturers might choose not to make 128-core CPUs by the year 2012 is that there won't be enough software to take advantages of that degree of parallelism. (Even with today's transistor count, you could put 100 Pentium-class processors on a single die.) Moreover, programming languages which have built-in support for scaling to many processors are going to win in the long run, and programming languages that don't have strong, built-in concurrency support will gradually fade away. The reason for this is simple: People don't like using applications which don't get faster when they buy a better computer. Now, on the server side, all of this is a solved problem. We have a very robust model for handling hundreds of requests in parallel, and a very strong model for dealing with shared state. This model is called "transactions" or more formally ACID. On the client side, however, things are much worse. As he puts it, locks are the best we have, and locks suck. Locks are not "logically composable", in the sense that you can't simply take two pieces of software that were written independently and glue them together unless they share a common locking architecture. This ruins a large part of the power of modern software, which is the ability to compose software out of smaller, independently-written components. On the client side, you have heterogeneous components accessing shared state at high bandwidth with myriads of pointer-based references to that shared state. Add concurrency, and what you end up with is a system in which it is impossible to make logical inferences about the behavior of the system. (And before you ask - yes, functional languages - and their drawbacks - are addressed in the talk. The same is true for 'lock-free programming' and other techniques du jour.) Now, in the talk he proposes some strategies for building concurrency into the language so that client-side programming can be done in a way that isn't rocket science. We won't be able to get rid of locks, he claims, but with the right language support at least we'll be able to reason about them, and maybe push them off to the side so that they mostly stay in their corner. In any case, what I would like to open up is a discussion of how these ideas might influence the design of the Python language. One thing that is important to understand is that I'm not talking about "automatic parallelization" where the compiler automatically figures out what parts can be parallelized. That would require so much change to the Python language as to make it no longer Python. Nor am I talking about manually creating "thread" objects, and having to carefully place protections around any shared state. That kind of parallelism is too low-level, too coarse-grained, and too hard to do, even for people who think they know how to write concurrent code. I am not even necessarily talking about changing the Python language (although certainly the internals of the interpreter will need to be changed.) The same language can be used to describe the same kinds of problems and operations, but the implications of those language elements will change. This is analogous to the fact that these massively multicore CPUs 10 years from now will most likely be using the same instruction sets as today - but that does not mean that programming as a discipline will anything like what it is now. As an example of what I mean, suppose the Python 'with' statement could be used to indicate an atomic transaction. Any operations that were done within that block would either be committed or rolled back. This is not a new idea - here's an interesting paper on it: http://www-sal.cs.uiuc.edu/~zilles/tm.html I'm sure that there's a lot more like that out there. However, there is also a lot of stuff out there that is *superficially* similar to what I am talking about, and I want to make the distinction clear. For example, any discussion of concurrency in Python will naturally raise the topic of both IPython and Stackless. However, IPython (from what I understand) is really about distributed computing and not so much about fine-grained concurrency; And Stackless (from what I understand) is really about coroutines or continuations, which is a different kind of concurrency. Unless I am mistaken (and I very well could be) neither of these are immediately applicable to the problem of authoring Python programs for multi-core CPUs, but I think that both of them contain valuable ideas that are worth discussing. Now, I will be the first to admit that I am not an expert in these matters. Don't let my poor, naive ideas be a limit to what is discussed. What I've primarily tried to do in this posting is to get your attention and convince you that this is important and worth talking about. And I hope to be able to learn something along the way. My overall goal here is to be able to continue writing programs in Python 10 years from now, not just as a hobby, but as part of my professional work. If Python is able to leverage the power of the CPUs that are being created at that time, I will be able to make a strong case for doing so. On the other hand, if I have a 128-core CPU on my desk, and Python is only able to utilize 1/128th of that power without resorting to tedious calculations of race conditions and deadlocks, then its likely that my Python programming will be relegated to the role of a hobby. -- Talin [View Less]

10 33

from __future__ import function_annotations
by Tobias Ivarsson April 11, 2007

April 11, 2007

I am just curiously wondering about the plans for introducing function annotations (PEP 3107). I could not find any information about this in the PEP, neither when I searched the mail archives. The way I see it this feature could be quite interesting to introduce as early as possible since I believe that there are quite a few tools that could benefit from this today. I could for example see Jython using function annotations for declaring methods that are supposed to be accessible from java code.… [View More]

5 5