Mailman 3 September 2007 - Python-ideas

Ideas towards GIL removal
by Greg Ewing July 18, 2008

July 18, 2008

I've been thinking about some ideas for reducing the amount of refcount adjustment that needs to be done, with a view to making GIL removal easier. 1) Permanent objects In a typical Python program there are many objects that are created at the beginning and exist for the life of the program -- classes, functions, literals, etc. Refcounting these is a waste of effort, since they're never going to go away. So perhaps there could be a way of marking such objects as "permanent" or "immortal". … [View More]

8 19

Enhance reload
by Joseph Maurer Oct. 1, 2007

Oct. 1, 2007

Ok, my idea of a temporary "replace" attribute wrapped around a reload-like function is not a good idea for Python given that things can be dynamically added to modules at any time. --------------------------------------- Here is my original high level design: The way I implemented this feature in Slick-C is with indirection. In Python terms, this means that a separate data structure that isn't reference counted holds the method/function object data. The method/function object is changed to … [View More]just contain a pointer to it. The data structure which holds all method/function data should probably be a non-reference counted dictionary. When a function is deleted, it's name remains in the dictionary but the entry needs to be changed to indicate that it is "null/invalid". When a deleted function is called, an exception should be raised. Adding a function/method means replacing the data in the dictionary. This type of implementation is simple. There's an insignificant amount of overhead on a function/method call (i.e. instead of "func->data" you have "func=*pfunc;if ( func->isInvalid() ) throw exception; else func->data" ). Technically this algorithm leaks memory since deleted functions/methods are never removed. My response is who cares. When the interpreter cleanup everything function is called, you simple deallocate everything in the dictionary. --------------------------------- Instead of a temporary "replace" attribute wrapped into a reload-like call, how about giving modules a user settable "replace" attribute. At any time, the user can set/reset this attribute. This would specify how the user wanted functions/methods processed. Always added or always replaced. The "replace" attribute would likely need to be pass through to function objects, class objects, and method objects. For the macro language scenarios, I would just mark every module that got loaded with this attribute. The proposed implementation I have given is intended to by very "file" oriented (which maps to a Python module). Would this work in the current code base? I'm assuming the following: When a function is added/executed, the module structure is accessible. When a class is added (i.e. class myclass), the module structure is accessible. When a method is added/executed, at least the class structure is accessible? I hope you see where I'm going here. The executed "class myclass" code which defines a new class can copy the module "replace" attribute. the executed "def myfunction" code which defines a new method can copy the class "replace" attribute. The function/method object structure could remain the same except for the addition of a new function/method pointer member. The additional code for a function call would look like this: // Did this function get defined in "replace" mode? if ( func->doReplace() ) { // For this one, use the indirect pointer and not the other member data. func= func->pfunc; if ( !func->isValid() ) { throw exception. // Python exeception, not C++ exception return here... } } // Now do what we used to do Given the OO nature of Python, a separate function/method type for a replacable function/method could be defined but I suspect it isn't worth the effort. The above psuedo code is very efficient "doReplace" would probably be just a boolean/int member. The "isValid" called would be efficient as well. One thing my proposed implementation does not cover is adding new data members to a class. I think it is acceptable for this not to be handled. Please shot this down this high level implementation if it won't work in the current code base. Also, what does everyone think about the idea of some sort of "replace" attribute for the module? How should it get set? "import module; module.replace=1". I'm probably showing a little lack of knowledge here. Teach me and I'll get it. ____________________________________________________________________________________ Pinpoint customers who are looking for what you sell. http://searchmarketing.yahoo.com/ [View Less]

2 1

Enhance reload
by Joseph Maurer Oct. 1, 2007

Oct. 1, 2007

Greg Ewin wrote: > What you suggest sounds like it ought to be possible, > at first sight, since Python function objects are already > containers with a reference to another object that > holds the function's code. > The problem will be figuring out *when* you're redefining > a function, because the process of loading a module is a > very dynamic one in Python. Defining functions and classes > is done by executing code, not by statically analysing > declarations as a … [View More]

2 1

Enhance reload
by Joseph Maurer Sept. 30, 2007

Sept. 30, 2007

I'm still new to the technical abilities of Python so help me if I misunderstand the current capabilities. I'd like to see the reload feature of Python enhanced so it can replace the methods for existing class instances, references to methods, and references to functions. Here's the scenario. Let's say you want to use Python as a macro language. Currently, you can bind a Python function to a key or menu (better do it by name and not reference). That's what most apps need. However, an … [View More]

5 4

Enhance reload
by Joseph Maurer Sept. 29, 2007

Sept. 29, 2007

I'm glad to hear it isn't a matter of whether it was useful or not. The way I implemented this feature in Slick-C is with indirection. In Python terms, this means that a separate data structure that isn't reference counted holds the method/function object data. The method/function object is changed to just contain a pointer to it. The data structure which holds all method/function data should probably be a non-reference counted dictionary. When a function is deleted, it's name remains in the … [View More]

3 2

Removing the del statement
by George Sakkis Sept. 28, 2007

Sept. 28, 2007

I guess this has very few to zero chances of being considered, even for Python 3, but this being python-ideas I guess it's ok to bring it up. IMO the del statement is one of the relatively few constructs that stick out like a sore thumb. For one thing, it is overloaded to mean three different things: 1) del x: Remove x from the current namespace 2) del x[i]: Equivalent to x.__delitem__(i) 3) del x.a: Equivalent to x.__delattr__('a') and delattr(x,'a') Here I am mostly arguing for removing the … [View More]

5 4

Exploration PEP : Concurrency for moderately massive (4 to 32 cores) multi-core architectures
by Clark Maurer Sept. 27, 2007

Sept. 27, 2007

Hello, I've been following this discussion. My thoughts mostly reiterate what has already been said. There's no way to get rid of the GIL without significantly effecting single threaded performance. IMO getting rid of the GIL would require writing a mark-and-sweep algorithm. To improve performance you can do incremental (threaded) marking and detect page faults so that modified pages can be rescanned for references. The Boehm garbage collector does this (I think) but Python would need … [View More]

2 1

Re: [Python-ideas] Calling a function of a list without accumulating results
by Terry Jones Sept. 27, 2007

Sept. 27, 2007

Hi Arnaud | If you want to do it like this, why not do it explicitly: | | def exhaust(iterable): | for i in iterable: pass | | Then you can write: | | exhaust(f(x) for x in mylist) Thanks - that's nice. It also gives me the generality I wanted, which was the ability to use the full LC/genexp "for..." syntax, which I should have emphasized more, including in the subject of the thread. Terry

1 0

Re: [Python-ideas] Calling a function of a list without accumulating results
by Terry Jones Sept. 27, 2007

Sept. 27, 2007

Hi Greg | The way things are, there is only one coding style for when you don't want | the results. You're suggesting the addition of another one. That *would* be | un-Pythonic. But the same remark could be made about using a list and writing explicit loops to accumulate results, and the later addition of list comprehensions. Wasn't that un-Pythonic for the same reason? Terry

1 0

Calling a function of a list without accumulating results
by Terry Jones Sept. 27, 2007

Sept. 27, 2007

What's the most compact way to repeatedly call a function on a list without accumulating the results? While I can accumulate results via a = [f(x) for x in mylist] or with a generator, there doesn't seem to be a way to do this without accumulating the results. I guess I need to either use the above and ignore the result, or use for x in mylist: f(x) I run into this need quite frequently. If I write [f(x) for x in mylist] with no assignment, will Python notice that I … [View More]

7 6