Nick Coghlan schrieb:

Actually, I believe 3.0 already took a big step towards allowing this by changing the way modules are initialised.

You are believing correctly. Martin has designed and implemented a nicely working API to store extension module data per interpreter state. For now interpreter states are used for sub interpreters only. http://www.python.org/dev/peps/pep-3121/ Christian

Thomas Heller wrote:

Christian Heimes schrieb:

...

Nick Coghlan schrieb:

...

Actually, I believe 3.0 already took a big step towards allowing this by changing the way modules are initialised. You are believing correctly. Martin has designed and implemented a nicely working API to store extension module data per interpreter state. For now interpreter states are used for sub interpreters only.

http://www.python.org/dev/peps/pep-3121/

But the extension modules still have to changed to use this mechanism, right?

Yep, but at least it's *possible* now. With 2.x, it isn't possible for an extension module to support subinterpreters properly, even if they want to. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------

2008/12/17 Greg Ewing :

Nick Coghlan wrote:

...

Actually, I believe 3.0 already took a big step towards allowing this by changing the way modules are initialised.

It's a step, but I wouldn't call it a big one. There are many other problems to be solved before fully independent interpreters are possible.

Do you know if these remaining problems are listed anywhere? AIUI, certain software (for example mod_python) has been using multiple interpreters for a long while now - admittedly not without issues, but certainly enough to imply that multiple interpreters are at least "possible" - although not perfect. Experience with such software would probably be a great guide to where the issues exist. Maybe a page on the Python Wiki, or a FAQ entry, would be useful here. If only to make things explicit, and clear up some of the FUD around multiple interpreters. Paul.

Greg Ewing wrote:

Paul Moore wrote:

...

Do you know if these remaining problems are listed anywhere?

There was a big discussion about this in comp.lang.python not long ago. Basically all the built-in types and constants are shared between interpreters, which means you still need a GIL to stop different interpreters stepping on each other's toes.

That kind of thing is under the core's control though - the 2.x module initialisation problem means that you can't write a multiple interpreter friendly extension module even if you want to. The new per-interpreter state mechanism could also be used internally by the core to duplicate some of that global state for each new interpreter. I see the introduction of the interpreter specific state mechanism as a big step because it provides an underlying mechanism that makes the problem solvable *in principle* through a combination of per-interpreter state and finer grained shared locking, making it just a practical implementation problem to see if that can be done without adversely impacting single interpreter performance. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------

Paul Moore schrieb:

OK, but how close is it to providing isolation for threads running under the control of the GIL? I'm thinking of something along the lines of an in-process version of fork(), which spawns a new interpreter and runs the 2 interpreters as threads, still using the GIL to enforce serialisation, but otherwise independent. I believe that Perl uses this model for its "interpreter threads" implementation.

How is your idea different from subinterpreters? Today you can have multiple subinterpreters inside a single process. Each subinterpreter has its own state and can see only its own objects. Christian

On 12/12/2008 11:52 AM, Lennart Regebro wrote:

The use of threads for load balancing should be discouraged, yes. That is not what they are designed for. Threads are designed to allow blocking processes to go on in the background without blocking the main process.

It seems that most programmers with Java or Windows experience don't understand this; hence the ever lasting GIL debate. With multiple interpreters - one interpreter per thread - this could still be accomplished. Let one interpreter block while another continues to work. Then the result of the blocking operation is messaged back. Multi-threaded C libraries could be used the in same way. But there would be no need for a GIL, because each interpreter would be a single-threaded compartment. .NET have something similar in what is called 'appdomains'. I am not suggesting removal of threads but rather the Java threading model. I just think it is a mistake to let multiple OS threads touch the same interpreter. Sturla Molden

Lennart Regebro wrote:

On Fri, Dec 12, 2008 at 02:13, Sturla Molden wrote:

...

I genuinely think the use of threads should be discouraged. It leads to code that are full of bugs and difficult to maintain - race conditions, deadlocks, and livelocks are common pitfalls.

The use of threads for load balancing should be discouraged, yes. That is not what they are designed for. Threads are designed to allow blocking processes to go on in the background without blocking the main process. This, they are very useful for. Removing thread support would therefore be a very big mistake. It's needed, it has it's uses, just not the one *you* want.

That's an interesting assertion about what threads were designed for. Do you have anything to back it up? Michael -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog

If I remember correctly (when threading was invented in the mid-1980s) threads were originally described as "lightweight processes". The perceived advantage at the time was the ability to have multiple threads of control with shared memory: this was much faster than the available inter-process communication mechanisms. On a single-processor computer synchronization was much less of a problem. regards Steve Guido van Rossum wrote:

Yes, this is what threads were designed for. As an abstraction to have multiple "threads of control" on a *single* processor (in a single process). The whole multi-core business came decades later. (Classic multi-processors have something called threads too, but they, too, came later than the original single-core-single-CPU thread concept, and often threads on those systems have properties that don't match how threads work on modern multi-core CPUs.)

On Sat, Dec 13, 2008 at 5:32 AM, Michael Foord wrote:

...

Lennart Regebro wrote:

...

On Fri, Dec 12, 2008 at 02:13, Sturla Molden wrote:

...

I genuinely think the use of threads should be discouraged. It leads to code that are full of bugs and difficult to maintain - race conditions, deadlocks, and livelocks are common pitfalls.

The use of threads for load balancing should be discouraged, yes. That is not what they are designed for. Threads are designed to allow blocking processes to go on in the background without blocking the main process. This, they are very useful for. Removing thread support would therefore be a very big mistake. It's needed, it has it's uses, just not the one *you* want.

That's an interesting assertion about what threads were designed for. Do you have anything to back it up?

-- Steve Holden +1 571 484 6266 +1 800 494 3119 Holden Web LLC http://www.holdenweb.com/

On Sat, Dec 13, 2008 at 9:13 AM, Christian Heimes wrote:

Steve Holden schrieb:

...

If I remember correctly (when threading was invented in the mid-1980s) threads were originally described as "lightweight processes". The perceived advantage at the time was the ability to have multiple threads of control with shared memory: this was much faster than the available inter-process communication mechanisms. On a single-processor computer synchronization was much less of a problem.

Initially one of Java's main target platforms were set-top boxes. Back in the 90ties set-top boxes had limited hardware and dumb processors. Most of the boxes had no MMU and so didn't support multiple processes. Threads were the easiest way to have some kind of concurrency.

Just let's not rewrite history and believe Java invented threads. They were around well before that.

Back in those days threads were the only solution for concurrency but today - about 15 years later with powerful processors even in cheap mobile phones - people are still indoctrinated with the same philosophy ...

It's not so much indoctrination. Threads are a useful tool. The problem is that some people perceive threads as the *only* tool. There's a whole spectrum of tools, from event handling to multiple processes, and they don't all solve the same problem. (I guess it doesn't help that the word process is given new meanings by some languages.) -- --Guido van Rossum (home page: http://www.python.org/~guido/)

On Dec 13, 2008, at 5:47 PM, Martin v. Löwis wrote:

They were originally invented in 1965, on Multics (1970) they were used to perform compilation in the background. When Unix came along, it *added* address space separation, introducing what is now known as processes.

Yes, and a lot of the subsequent interest in threads came due to the historically debilitating overhead of fork() on some important Unices, notably Solaris. -- Ivan Krstić | http://radian.org