Mailman 3 May 2007 - Python-ideas

Re: [Python-ideas] [Python-3000] the future of the GIL
by Leonardo Santagada May 9, 2007

May 9, 2007

Em 09/05/2007, às 06:07, Giovanni Bajo escreveu: > On 07/05/2007 7.36, Josiah Carlson wrote: > >> By going multi-process rather than multi-threaded, one generally >> removes >> shared memory from the equasion. Note that this has the same >> effect as >> using queues with threads, which is generally seen as the only way of >> making threads "easy". If one *needs* shared memory, we can >> certainly >> create an mmap-based shared memory subsystem with fine-grained object >> locking, or emulate it via a server process as the processing package >> has done. >> >> Seriously, give the processing package a try. It's much faster >> than one >> would expect. > > I'm fully +1 with you on everything. > > And part of the fact that we have to advocate this is because > Python has > always had pretty good threading libraries, but not processing > libraries; > actually, Python does have problems at spawning processes: the whole > popen/popen2/subprocess mess isn't even fully solved yet. > > One thing to be said, though, is that using multiple processes > cause some > headaches with frozen distributions (PyInstaller, py2exe, etc.), > like those > usually found on Windows, specifically because Windows does not > have fork(). > > The processing module, for instance, doesn't take this problem into > account at > all, making it worthless for many of my real-world use cases. > -- > Giovanni Bajo Another problem is that althought people like the idea of processing more, the only module on python stdlib is thread and Threading... So why not include processing in stdlib and in both thread and Threading give pointers to its use? This would probably diminish the problem with newcomers on the language that hate the GIL because they don't know better (and also because threading is supported out of the box on python). Yes I know there is fork, but still it is not as useable and pythonic as processing. -- Leonardo Santagada santagada(a)gmail.com

1 0

Re: [Python-ideas] [Python-3000] the future of the GIL
by Leonardo Santagada May 9, 2007

May 9, 2007

Em 09/05/2007, às 04:26, James Y Knight escreveu: > On May 7, 2007, at 1:58 PM, Guido van Rossum wrote: >> As C doesn't have an atomic increment nor an atomic >> decrement-and-test, the INCREF and DECREF macros sprinkled throughout >> the code (many thousands of them) must be protected by some lock. > > I've been intently ignoring the rest of the thread (and will continue > to do so), but, to respond to this one particular point... > > This just isn't true. Python can do an atomic increment in a fast > platform specific way. It need not restrict itself to what's > available in C. (after all, *threads* aren't available in C....) > > Two implementations of note: > > 1) gcc 4.1 has atomic operation builtins: > http://gcc.gnu.org/onlinedocs/gcc-4.1.2/gcc/Atomic- > Builtins.html#Atomic-Builtins > > 2) There's a pretty damn portable library which provides these > functions for what looks to me like pretty much all CPUs anyone would > use, under Linux, Windows, HP/UX, Solaris, and OSX, and has a > fallback to using pthreads mutexes: > > http://www.hpl.hp.com/research/linux/atomic_ops/index.php4 > http://packages.debian.org/stable/libdevel/libatomic-ops-dev > > > It's quite possible the overhead of GIL-less INCREF/DECREF is still > too high even with atomic increment/decrement primitives, but AFAICT > nobody has actually tried it. So saying GIL-less operation for sure > has too high of an overhead unless the refcounting GC is replaced > seems a bit premature. > > James This seems like a good idea... how about moving this discussion to python-ideas? -- Leonardo Santagada santagada(a)gmail.com

1 0

subclassing standard module
by Aaron Brady May 9, 2007

May 9, 2007

This on module inspect. I'd like to subclass inspect.BlockFinder, but inspect.getblock uses it directly. Should we encapsulate this?

1 0

PEP for executing a module in a package containing relative imports
by Brett Cannon May 7, 2007

May 7, 2007

Some of you might remember a discussion that took place on this list about not being able to execute a script contained in a package that used relative imports (read the PEP if you don't quite get what I am talking about). The PEP below proposes a solution (along with a counter-solution). Let me know what you think. I especially want to hear which proposal people prefer; the one in the PEP or the one in the Open Issues section. Plus I wouldn't mind suggestions on a title for this PEP. =) ------------------------------------------- PEP: XXX Title: XXX Version: $Revision: 52916 $ Last-Modified: $Date: 2006-12-04 11:59:42 -0800 (Mon, 04 Dec 2006) $ Author: Brett Cannon Status: Draft Type: Standards Track Content-Type: text/x-rst Created: XXX-Apr-2007 Abstract ======== Because of how name resolution works for relative imports in a world where PEP 328 is implemented, the ability to execute modules within a package ceases being possible. This failing stems from the fact that the module being executed as the "main" module replaces its ``__name__`` attribute with ``"__main__"`` instead of leaving it as the actual, absolute name of the module. This breaks import's ability to resolve relative imports from the main module into absolute names. In order to resolve this issue, this PEP proposes to change how a module is delineated as the module that is being executed as the main module. By leaving the ``__name__`` attribute in a module alone and setting a module attribute named ``__main__`` to a true value for the main module (and thus false in all others), proper relative name resolution can occur while still having a clear way for a module to know if it is being executed as the main module. The Problem =========== With the introduction of PEP 328, relative imports became dependent on the ``__name__`` attribute of the module performing the import. This is because the use of dots in a relative import are used to strip away parts of the calling module's name to calcuate where in the package hierarchy a relative import should fall (prior to PEP 328 relative imports could fail and would fall back on absolute imports which had a chance of succeeding). For instance, consider the import ``from .. import spam`` made from the ``bacon.ham.beans`` module (``bacon.ham.beans`` is not a package itself, i.e., does not define ``__path__``). Name resolution of the relative import takes the caller's name (``bacon.ham.beans``), splits on dots, and then slices off the last n parts based on the level (which is 2). In this example both ``ham`` and ``beans`` are dropped and ``spam`` is joined with what is left (``bacon``). This leads to the proper import of the module ``bacon.spam``. This reliance on the ``__name__`` attribute of a module when handling realtive imports becomes an issue with executing a script within a package. Because the executing script is set to ``'__main__'``, import cannot resolve any relative imports. This leads to an ``ImportError`` if you try to execute a script in a package that uses any relative import. For example, assume we have a package named ``bacon`` with an ``__init__.py`` file containing:: from . import spam Also create a module named ``spam`` within the ``bacon`` package (it can be an empty file). Now if you try to execute the ``bacon`` package (either through ``python bacon/__init__.py`` or ``python -m bacon``) you will get an ``ImportError`` about trying to do a relative import from within a non-package. Obviously the import is valid, but because of the setting of ``__name__`` to ``'__main__'`` import thinks that ``bacon/__init__.py`` is not in a package since no dots exist in ``__name__``. To see how the algorithm works, see ``importlib.Import._resolve_name()`` in the sandbox [#importlib]_. Currently a work-around is to remove all relative imports in the module being executed and make them absolute. This is unfortunate, though, as one should not be required to use a specific type of resource in order to make a module in a package be able to be executed. The Solution ============ The solution to the problem is to not change the value of ``__name__`` in modules. But there still needs to be a way to let executing code know it is being executed as a script. This is handled with a new module attribute named ``__main__``. When a module is being executed as a script, ``__main__`` will be set to a true value. For all other modules, ``__main__`` will be set to a false value. This changes the current idiom of:: if __name__ == '__main__': ... to:: if __main__: ... The current idiom is not as obvious and could cause confusion for new programmers. The proposed idiom, though, does not require explaining why ``__name__`` is set as it is. With the proposed solution the convenience of finding out what module is being executed by examining ``sys.modules['__main__']`` is lost. To make up for this, the ``sys`` module will gain the ``main`` attribute. It will contain a string of the name of the module that is considered the executing module. A competing solution is discussed in `Open Issues`_. Transition Plan =============== Using this solution will not work directly in Python 2.6. Code is dependent upon the semantics of having ``__name__`` set to ``'__main__'``. There is also the issue of pre-existing global variables in a module named ``__main__``. To deal with these issues, a two-step solution is needed. First, a Py3K deprecation warning will be raised during AST generation when a global variable named ``__main__`` is defined. This will help with the detection of code that would reset the value of ``__main__`` for a module. Without adding a warning when a global variable is injected into a module, though, it is not fool-proof. But this solution should cover the vast majority of variable rebinding problems. Second, 2to3 [#2to3]_ will gain a rule to transform the current ``if __name__ == '__main__': ...`` idiom to the new one. While it will not help with code that checks ``__name__`` outside of the idiom, that specific line of code makes up a large proporation of code that every looks for ``__name__`` set to ``'__main__'``. Open Issues =========== A counter-proposal to introducing the ``__main__`` attribute on modules was to introduce a built-in with the same name. The value of the built-in would be the name of the module being executed (just like the proposed ``sys.main``). This would lead to a new idiom of:: if __name__ == __main__: ... The perk of this idiom over the one proposed earlier is that the general semantics does not differ greatly from the current idiom. The drawback is that the syntactic difference is subtle; the dropping of quotes around "__main__". Some believe that for existing Python programmers bugs will be introduced where the quotation marks will be put on by accident. But one could argue that the bug would be discovered quickly through testing as it is a very shallow bug. The other pro of this proposal over the earlier one is the alleviation of requiring import code to have to set the value of ``__main__``. By making it a built-in variable import does not have to care about ``__main__`` as executing the code itself will pick up the built-in ``__main__`` itself. This simplies the implementation of the proposal as it only requires setting a built-in instead of changing import to set an attribute on every module that has exactly one module have a different value (much like the current implementation has to do to set ``__name__`` in one module to ``'__main__'``). References ========== .. [#2to3] 2to3 tool (http://svn.python.org/view/sandbox/trunk/2to3/) [ViewVC] .. [#importlib] importlib (http://svn.python.org/view/sandbox/trunk/import_in_py/importlib.py?view=mar…) [ViewVC] Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

14 58

Should function attributes be properties?
by Joel Bender May 3, 2007

May 3, 2007

I was expecting this to work: class myprop(property): def __get__(self, obj, cls): print "get", obj, cls def __set__(self, obj, value): print "set", obj, value def f(x): print 'f' f.p f.p = myprop() So it comes out like this: >>> f(2) f get <function f at 0x61270> None Is this a bug? Or was it not in the scope of what properties would be used for? Is this PEP material? Joel

3 2

fixing mutable default argument values
by Chris Rebert May 2, 2007

May 2, 2007

If A.M. Kuchling's list of Python Warts is any indication, Python has removed many of the warts it once had. However, the behavior of mutable default argument values is still a frequent stumbling-block for newbies. It is also present on at least 3 different lists of Python's deficiencies ([0][1][2]). Example of current, unintuitive behavior (snipped from [0]): >>> def popo(x=[]): ... x.append(666) ... print x ... >>> popo() [666] >>> popo() [666, 666] >>> popo() [666, 666, 666] Whereas a newbie with experience with immutable default argument values would, by analogy, expect: >>> popo() [666] >>> popo() [666] >>> popo() [666] In scanning [0], [1], [2], and other similar lists, I have only found one mediocre use-case for this behavior: Using the default argument value to retain state between calls. However, as [2] comments, this purpose is much better served by decorators, classes, or (though less preferred) global variables. Other uses are alluded to be equally esoteric and unpythonic. To work around this behavior, the following idiom is used: def popo(x=None): if x is None: x = [] x.append(666) print x However, why should the programmer have to write this extra boilerplate code when the current, unusual behavior is only relied on by 1% of Python code? Therefore, I propose that default arguments be handled as follows in Py3K: 1. The initial default value is evaluated at definition-time (as in the current behavior). 2. That in a function call where the caller has not specified a value for an optional argument, Python calls copy.deepcopy(initial_default_value), and fills in the optional argument with the resulting value. This is fully backwards-compatible with the aforementioned workaround, and removes the need for the it, allowing one to write the first, simpler definition of popo(). Comments? - Chris Rebert [0] 10 Python pitfalls (http://zephyrfalcon.org/labs/python_pitfalls.html) [1] Python Gotchas (http://www.ferg.org/projects/python_gotchas.html#contents_item_6) [2] When Pythons Attack (http://www.onlamp.com/pub/a/python/2004/02/05/learn_python.html?page=2)

9 25

ordered dict
by Mathias Panzenböck May 1, 2007

May 1, 2007

Some kind of ordered dictionary would be nice to have in the standard library. e.g. a AVL tree or something like that. It would be nice so we can do things like that: for value in tree[:end_key]: do_something_with(value) del tree[:end_key] A alternative would be just to sort the keys of a dict but that's O( n log n ) for each sort. Depending on what's the more often occurring case (lookup, insert, get key-range, etc.) a other kind of dict object would make sense. What do you think? -panzi

5 18