Parrot -- should life imitate satire?

The 2001 O'Reilly Open Source Convention, was, as usual, a very stimulating event and a forum for a lot of valuable high-level conversations between the principal developers of many open-source projects. Many of you know that I maintain friendly and relatively close relations with a number of senior Perl hackers, including both Larry Wall himself and others like Chip Salzenberg, Randall Schwartz, Tom Christiansen, Adam Turoff, and more recently Simon Cozens (who lurks on this list these days). At OSCon I believe I got a pretty good picture of what the leaders of the Perl community are thinking and planning these days. They have definitely come out of the slump they were in a year ago -- there's a much-renewed sense of energy over there. I think their plans offer both the Perl and Python communities some large strategic opportunities. Specifically, I'm urging the Python community's leadership to seriously explore the possibility of helping make the Parrot hoax into a working reality. I have discussed this with Guido by phone, and though he is skeptical about such an implementation being actually possible, he also thinks the idea has tremendous potential and says he is willing to support it in public. The Perl people have blocked out an architecture for Perl 6 that envisages a new bytecode level, designed and implemented from scratch. They're very serious about this; I participated in some discussions of the bytecode design (and, incidentally, argued that the bytecode should emulate a stack rather than a register machine because the cost/speed disparities that justify register architectures in hardware don't exist in a software VM). The Perl people are receptive -- indeed, some of them are actively pushing -- the idea that their new bytecode should not be Perl-specific. Dan Sugalski, the current lead for the bytecode interpreter project, has named it Parrot. At the Perl 6 talk I attended, Chip Salzenberg speculated in public about possibly supporting a common runtime for Perl, Python, Ruby, and Intercal(!). One of the things that makes this an unprecedented opportunity is that the design of Perl 6 is not yet set in stone -- and Larry has already shown a willingness to move it in a Pythonward direction. Syntactically, Perl 5's -> syntax is going away to be replaced by a Python-like dot with implicit dereferencing (and Larry said in public this was Python's influence, not Java's). The languages have of course converged in other ways recently -- Python's new lexical scoping actually brings it closer to Perl "use strict" semantics. I believe the way is open for Python's leading technical people to be involved as co-equals in the design and implementation of the Parrot bytecode interpreter. I have even detected some willingness to use Python's existing bytecode as a design model for Parrot, and perhaps even portions of the Python interpreter codebase! One bold but possibly workable proposal would be to offer Dan and the Parrot project the Python bytecode interpreter as a base for the Parrot code, and then be willing to incorporate whatever (probably relatively minor) extensions are needed to support Perl 6's primitives. Following my conversation with Guido, I've put doing an architectural comparison of the existing Python and Perl bytecodes at the top of my priority list. I'm flying to Taipei tomorrow and will have a lot of hours on airplanes with my laptop to do this. Committing a common runtime with Perl would mean relinquishing exclusive design control of our bytecode level, but the Perl people seem themselves willing to give up *their* exclusive control to make this work. It is rather remarkable how respectful of Python they have become, and I can't emphasize enough that I think they are truly ready for us to come to the project as equal partners. (One important place where I think everybody understands the Python side of the force would clearly win out in a final Parrot design is in the extension-and-embedding facilities. Perl XS is acknowledged to be a nasty mess. My guess is the Perl guys would drop it like a hot rock for our stuff -- that would be as clear a win for them as co-opting Perl-style regexps was for us.) I think the benefits of a successful unification at the bytecode level, together with Larry's willingness to Pythonify the upper level of Perl 6 a bit, could be vast -- both for the Python community in particular and for scripting-language users in general. 1. Mixed-language programming in Perl and Python could become almost seamless, with all that implies for both languages getting the use of each others' libraries. 2. The prospects for getting decent Python compilation to native code would improve if both the Python and Perl communities were strongly motivated to solve the bytecode-compilation problem. 3. More generally, the fact remains that Perl's user/developer base is still much larger than ours. Successful unification would co-opt a lot of that energy for Python. Because the brain drain between Perl and Python is pretty much unidirectional in Python's favor (a fact even the top Perl hackers ruefully acknowledge), I don't think we need worry about being subsumed in that larger community either. I think there is a wonderful opportunity here for the Python and Perl developers to lead the open-source world. If we can do a good Parrot design together, I think it will be hard for the smaller scripting language communities to resist its pull. Ultimately, the common Parrot runtime could become the open-source community's answer -- a very competititive answer -- to the common runtime Microsoft is pushing for .NET. I think trying to make Parrot work would be worth some serious effort. -- <a href="http://www.tuxedo.org/~esr/">Eric S. Raymond</a> The Bible is not my book, and Christianity is not my religion. I could never give assent to the long, complicated statements of Christian dogma. -- Abraham Lincoln

Obviously, just as the new design is aiming at Perl 6, it would be aiming at Python 3. Nothing's impossible these days, so I am keeping an open mind. I expect that in addition to the bytecode, the entire runtime architecture would have to be shared though for this to make sense, and I'm not sure how easy that would be, even if Perl is willing to be flexible. Most of Python's run-time semantics are very carefully defined and shouldn't be changed in order to fit in the common runtime. I'm looking forward to Eric's comparison of the two run-time systems. (Eric, be sure to use a copy of 2.2a1 or the descr-branch -- *don't* use the CVS trunk.) --Guido van Rossum (home page: http://www.python.org/~guido/)

Guido van Rossum <guido@zope.com>:
What would the CVS magic invocation for that be? And...um...why? Has the bytecode changed significantly recently? -- <a href="http://www.tuxedo.org/~esr/">Eric S. Raymond</a> The spirit of resistance to government is so valuable on certain occasions, that I wish it always to be kept alive. It will often be exercised when wrong, but better so than not to be exercised at all. I like a little rebellion now and then. -- Thomas Jefferson, letter to Abigail Adams, 1787

What would the CVS magic invocation for that be?
cvs update -r descr-branch or cvs checkout -r descr-branch python/dist/src Or just download 2.2a1.
And...um...why? Has the bytecode changed significantly recently?
Not the bytecode, but the rest of the runtime has changed tremendously, and as I tried to explain over the phone, that has a big impact on reusability of the runtime. The bytecode engine cannot be considered independent from the rest of the runtime. --Guido van Rossum (home page: http://www.python.org/~guido/)

Guido van Rossum <guido@zope.com>:
Or just download 2.2a1.
It's cool. My local installation is from 2.2.a0. I'll update.
OK, let's try to factor this design problem. Let's suppose, for the sake of the design discussion, that we can make the type ontologies of the Perl and Python bytecode match up. (Note: making the type ontologies of the two bytecodes match is not the same problem as making the type ontologies of the *languages* match up. It should be rather simpler because a lot of the differences between, e.g., class semantics can probably be compiled away. Not a trivial problem, but humor me.) Let's further suppose that we have a callout mechanism from the Parrot interpreter core to the Perl or Python runtime's C level that can pass out Python/Perl types and return them. Given these two premises, what other problems are there? I can see one: garbage collection. What others are there? -- <a href="http://www.tuxedo.org/~esr/">Eric S. Raymond</a> An armed society is a polite society. Manners are good when one may have to back up his acts with his life. -- Robert A. Heinlein, "Beyond This Horizon", 1942

"ESR" == Eric S Raymond <esr@thyrsus.com> writes:
ESR> Let's suppose, for the sake of the design discussion, that we ESR> can make the type ontologies of the Perl and Python bytecode ESR> match up. What is a type ontology? The definition of ontology I'm familiar with is too broad to be useful in understanding what you're getting at. I've never heard the technical term "type ontology". ESR> (Note: making the type ontologies of the two bytecodes match is ESR> not the same problem as making the type ontologies of the ESR> *languages* match up. It should be rather simpler because a ESR> lot of the differences between, e.g., class semantics can ESR> probably be compiled away. Not a trivial problem, but humor ESR> me.) If I guess at what you mean-- a fuzzy notion that the underlying type system can support both languages-- then I submit that most of the hard problems are indeed here. ESR> Let's further suppose that we have a callout mechanism from the ESR> Parrot interpreter core to the Perl or Python runtime's C level ESR> that can pass out Python/Perl types and return them. Not quite sure what you mean ehre. ESR> Given these two premises, what other problems are there? ESR> I can see one: garbage collection. What others are there? I think you mean memory management in general, not just GC. Others: thread model, interpreter management (such as creating embedded interpreter objects). Jeremy

Jeremy Hylton <jeremy@zope.com>:
I first heard it in connection with cross-language RPC. The "type ontology" of a language or protocol is its implicit theory of what kinds of things there are in the universe. It's actually a pretty reasonable specialization of the term "ontology" in philosophy. -- <a href="http://www.tuxedo.org/~esr/">Eric S. Raymond</a> A man who has nothing which he is willing to fight for, nothing which he cares about more than he does about his personal safety, is a miserable creature who has no chance of being free, unless made and kept so by the exertions of better men than himself. -- John Stuart Mill, writing on the U.S. Civil War in 1862

On Mon, Jul 30, 2001 at 05:18:31AM -0400, Eric S. Raymond wrote:
Let's suppose, for the sake of the design discussion, that we can make the type ontologies of the Perl and Python bytecode match up.
I'm afraid I'll have to side with Jeremy when I say, "What?"
Given these two premises, what other problems are there?
I can see one: garbage collection. What others are there?
As a midnight braindump: What about Perl's 'dynamic' (or 'really JIT') compilation ? The incessant weak typing -- would this be part of the Perl side of Parrot, or part of the Parrot types ? The differences in the regex engine; in Python, regular expressions are optional. Also, the Perl engine has some features SRE hasn't, yet, and vice versa (last I checked, Perl's regexps didn't do unicode or named groups.) And what about Perl's 'Taint' mode ? I don't see how you can emulate that ontop of the Parrot runtime, as it's a tag that gets carried into operations. And I won't even start with Perl's more archaic features, that change the whole working of the interpreter. You mentioned regular expressions as an upside for Python, from this 'merger'. Why is that ? We have a good regex engine, and it's tuned to Python's needs. Do we need 'regex literals' ? Why ? And why would we need a merger with Perl for that, anyway -- I've seen some arbitrary-type-literals suggestions come by in the last couple of days that would make it possible :-) -- Thomas Wouters <thomas@xs4all.net> Hi! I'm a .signature virus! copy me into your .signature file to help me spread!

Thomas Wouters <thomas@xs4all.net>:
Explained in public reply to Jeremy.
You mentioned regular expressions as an upside for Python, from this 'merger'. Why is that ?
No. I was referring to the fact that we have *already* coopted Perl's regexp design. -- <a href="http://www.tuxedo.org/~esr/">Eric S. Raymond</a> The end move in politics is always to pick up a gun. -- R. Buckminster Fuller

On Tue, Jul 31, 2001 at 01:24:32AM +0200, Thomas Wouters wrote:
Parrot types ? The differences in the regex engine; in Python, regular expressions are optional. Also, the Perl engine has some features SRE
If regex opcodes form part of the basic VM, would the main loop end up looking like the union of ceval.c and pypcre.c/_sre.c? The thought is too ghastly to contemplate, though a little part of me [*] would like to see it. --amk [*] "Taking it in its deepest sense, the shadow is the invisible saurian tail that man still drags behind him. Carefully amputated, it becomes the healing serpent of the mysteries. Only monkeys parade with it." C.G. Jung, in _The Integration of the Personality_. (1939)

Andrew Kuchling writes:
(perl guy speaking alert) The plan for perl6 is to implement the regular expression engine as opcodes. We feel this would be cleaner and faster than having the essentially separate module that we have right now. I think our current perl5 project manager was the one who said that we have no idea how inefficient our current RE engine is, because it's been "optimized" to the point where it's impossible to read. The core loop would just be the usual opcode dispatch loop ("call the function for the current operation, which returns the next operation"). The only difference is that some of the opcodes would be specific to RE matches. (I'm unclear on how much special logic RE opcodes involve--it may be possible to implement REs with the operations that regular language features like loops and tests require). Nat

On Mon, Jul 30, 2001 at 08:08:47PM -0700, Nathan Torkington wrote:
The big difference I see between regex opcodes and language opcodes is that regexes need to backtrack and language ones don't. Unless the idea is to compile a regex to actual VM code similar to that generated by Python/Perl code, but then wouldn't that sacrifice efficiency? --amk

At 08:01 AM 7/31/2001 -0400, Andrew Kuchling wrote:
Backtracking in regexes isn't all that much different from some of the things you do at the language level, and certainly it's a straightforward mapping of regexes onto a series of string & integer register manipulations with simple branch-and-test code, with perhaps some stack space thrown in for deeply nested regexes.
The idea is (currently, barring bad performance) to compile down to VM code similar to what Ptyhon or perl would generate. There's the potential to slow things down, which I'm rather nervous about. On the other hand, the sort of code that would get emitted for regexes maps very well onto the instruction sets of hardware CPUs, so you'd get a corresponding boost in speed once translated to machine code. (Though granted you'd need to be able to do the conversion on the fly, as the regex itself isn't guaranteed to be known at compile time) Dan --------------------------------------"it's like this"------------------- Dan Sugalski even samurai dan@sidhe.org have teddy bears and even teddy bears get drunk

Dan, Have you looked at the byte code used in SRE, Python's (mostly) Perl-compatible regex package? Worth a look! --Guido van Rossum (home page: http://www.python.org/~guido/)

At 11:46 PM 7/31/2001 -0400, Guido van Rossum wrote:
Have you looked at the byte code used in SRE, Python's (mostly) Perl-compatible regex package? Worth a look!
Nope, not yet, but I will. (You folks will end up making a Python programmer out of me yet, I expect... :) This and the C extension interface are the two big Python things I've got on my "must peer deeply into" list. Are there other places I should be concentrating on? (I'm poking about with the interpreter loop and associated code, but that's the sort of thing an expert with 20 minutes free will get me much farther than a week or two with the source) Dan --------------------------------------"it's like this"------------------- Dan Sugalski even samurai dan@sidhe.org have teddy bears and even teddy bears get drunk

On Tue, Jul 31, 2001 at 11:52:01PM -0400, Dan Sugalski wrote:
Nope, not yet, but I will. (You folks will end up making a Python programmer out of me yet, I expect... :)
Well, at least you know it's happening... most people don't realize until it's too late :)
What do you say I write up a longish email explaining quickly how Python objects work, and how the Python interpreter and extension code works with them ? I think it'd help with getting us talking about the same thing when we say 'interpreter' or 'type' :-) And the beauty of my being in a different timezone is that noone else is posting right now, so I can take the time without someone else doing it spread out over four posts in separate subthreads :) -- Thomas Wouters <thomas@xs4all.net> Hi! I'm a .signature virus! copy me into your .signature file to help me spread!

On Tue, Jul 31, 2001 at 11:52:01PM -0400, Dan Sugalski wrote:
[SRE] and the C extension interface are the two big Python things I've got on my "must peer deeply into" list.
I can't say much about SRE, but the C extension interface should be on anyone's "must peer deeply into" list, except that there isn't a real need to peer deeply into it :) Lets peer into it. I'm going to discuss the pre-2.2 state of affairs, mostly because I know jack shit about the 2.2-type-dichotomy-healing :) In case you aren't aware, we are currently into the 2.2 alphas, where this story is slightly different because you can now subclass types. If Guido or anyone else involved in the descr-branch wants to tune in and correct the explanation for Python 2.2, that's fine by me. As you might have heard, everything in Python is an object. This is true even for the C side of things. A Python object, any Python object, is essentially a struct like this: stuct PyExample { int ob_refcnt; /* reference count */ struct _typeobject *ob_type; /* type of this object */ /* type-specific data goes here */ } The common header is made available as a #define, so that we can add extra info in severe-debug mode (though this breaks binary compatibility, so it should not be done in production builds.) For instance, the struct to hold a (normal, unbounded) Python integer is typedef struct { PyObject_HEAD long ob_ival; } PyIntObject; The struct _typeobject pointer is a pointer to a Type object, which also acts as a vtable. It contains all the information about the type, including how to operate on it. The typeobject is itself also an object! Its type is the PyType type, which is its own type. The typeobject is actually what you get back when you do 'type(object)' in Python code. All this is also explained in Include/object.h, by the way. The type object holds things like the type name, how to print it, howmuch memory should be allocated, a couple of flags to implement binary compatibility, and a whole boatload of function pointers. The function pointers for operations are spread out over the type struct and several separate structs, to which the type struct points: tp_as_number, tp_as_sequence, and tp_as_mapping. Each of these can be NULL if the type doesn't implement the operations. Some of the operations that are defined on the type struct itself are get-attribute, set-attribute, comparison, get-hashvalue, convert-to-string and call-as-function. The PyNumberMethods struct (tp_as_number) contains arithmatic operations (such as adding, mulitplying, shifting, XORing, and in-place variations of each for use with augmented assignment), truth-value-testing, and converting to various other integer types. The PySequenceMethods struct (tp_as_sequence) contains sequence operations such as membership-test, item- and slice-retrieval and -assignment, and sequence concatenation and repetition. The latter two are somewhat of an odd duck, since they implement the same Python operation ("+" and "*") as the 'add' and 'multiply' methods from the PyNumberMethods struct. They're mainly so the Python runtime can properly handle "number * list" as well as "list * number" without attempting to convert list to an integer. Lastly there's PyMappingMethods. I'm not at all sure why this is a separate sturct, as the three operations it contains (length inquiry, item-retrieval and item-assignment) are also covered by the PySequenceMethods, but I suspect some kind of hysterical raisin crept in here :) To see this construct in action, take a look at Objects/intobject.c, in the Python source tree (doesn't really matter which version.) It defines a bunch of functions to do the arithmatic, fills a PyNumberMethods struct with the appropriate pointers (leaving unsupported ones NULL), and fills a new PyTypeObject with the right info for the "int" type. To contrast, the PyFunction type is nearly empty. It defines getattr and setattr, and a 'repr' function (convert to string). You can't compare it (it will always compare false to anything but itself), add it, index it, or get its length, but you can get attributes from it. And the average function has some interesting attributes: you can, from Python code and C code alike, easily get at things like the bytecode it consists of, the number of arguments it expects, the names of those arguments, etc. Each of those is, of course, a Python object in its own right :) Function objects are defined in Objects/funcobject.c, though the actual execution of function objects is done elsewhere. An 'instance' object is a slightly different case in this story (more so in the post-2.2 story, but I'm not sure howmuch more.) An instance object is a Python class, defined in Python code by the user/programmer. It can choose to implement any of the above operations by defining magically named methods on the class. For instance, to implement the 'getitem' protocol, it defines a function __getitem__, and to implement addition, __add__. The Type object for instance objects acts as a proxy for the Python class, though this does represent some problems: instance objects need to be treated in some areas of the interpreter (but this is almost certain to have been changed in 2.2.) But back to generic Python objects. Obviously, when writing C code, you don't want to manipulate Python objects by directly accessing all those layers of C structs all the time. So each builtin type defines useful C functions and macros to help. For instance, to create a new list, you call PyList_New(size). To test whether something is a list, PyList_Check(object). To index a list, PyList_GetItem(list, index), or a macro version of that, PyList_GET_ITEM. The macro version doesn't do type or boundary checks, so it should only be used when you already did those checks. Likewise, there is PyInt_AsLong (convert a PyInt to a C 'long') and PyInt_AS_LONG. But even that isn't that helpful in a dynamically typed language, since you have to know an object's type in order to call the type-specific functions. So we also have two more abstraction layers. The most abstract one is the Object layer. Basically, regardless of the type of object, you can always manipulate it using PyObject_*() functions. Some PyObject_ functions have a direct relation to Python code, others are only useful to C code. There are PyObject_ functions such as PyObject_IsTrue, PyObject_Compare, PyObject_GetAttr, PyObject_CallObject, etcetera. All these work regardless of the type. If the type (or instance) does not implement the operation, an exception is raised (see below.) The other layer is the slightly less abstract layer that differentiates in the same way as the PyNumberMethods, PySequenceMethods, PyMappingMethods structs. To add two objects, you call PyNumber_Add(o1, o2). To explicitly concatenate two objects, you call PySequence_Concat(o1, o2). All of these work for any type that defines the behaviour, including instance types. Note that calling one of these functions can end up executing arbitrary Python code, so they shouldn't be called in any time-critical situation :-) As a last part of the API, there is the reference counting. As I showed a couple of pages back, all objects contain a reference count. Operations that copy a reference should increment the reference count, using Py_INCREF, and deleting a reference should be done using Py_DECREF. The latter also does the deallocation and cleanup if the DECREF causes the refcount to drop to 0. All API functions that return a reference define whether they return a new reference, which the caller has to DECREF when throwing it away, or a borrowed reference, which shouldn't be DECREF'ed, or kept around without INCREFing it. Exceptions, by the way, are set by setting a global (or thread-local, in a threaded interpreter) variable to contain the Python exception object, and then returning an error flag (either NULL, -1 or 0, depending on the function) to the caller. All code should check all return values and always return an error value if a called function does so, unless it's prepared to handle the exception itself. To compare errors with a particular error 'class', there is PyErr_ExceptionMatches (which also handles 'subclasses' of exception types.) If the error value returned by a particular function is also a real possible value (such as the value returned by PyNumber_AsLong), there is PyErr_Occured() to see if it was a real exception, or a legitimate -1 value. So that's it for the API. All Python objects can be manipulated, created and destroyed this way. Most extension types provide their functionality either in builtin-operations (in the type object) or by providing methods that can be called (using PyObject_GetAttr and PyObject_CallObject), and only rarely need to export an explicit API of their own. (And as you probably know, exporting an API from a shared library to another shared library is... slightly tricky :) Now, the Python bytecode-interpreter itself is very small. All it does is interpret bytecode, and then call out to the API to make it do the actual work. It has some special knowledge about some types, for optimzation, and it does all the real exception handling and namespace handling (name lookup and such), but the actual manipulation of objects is left to the API. Even so, it has enough to do. In the case of exceptions, it has to search up the function for a 'catching' block (a try/except or try/finally statement), or break out of the function to let the error percolate to the caller. Namespaces are a complete story in their own right. Python has three distinct namespaces: the 'local' namespace inside a function (or 'code block'), the 'global' or 'module-level' namespace at the toplevel of a file, and the 'builtin' namespace which holds builtin functions and names. The search order is always local->global->builtin, though for code that is executed at the module level (not inside a function), local is the same as global. In Python, variables are defined by assignment. As a consequence, any variable that is assigned to in a code block, is local to that block. Any variable that isn't assigned to but is just referenced thus has to be defined in the global namespace. The compiler keeps track of this, and generates 'LOAD_FAST' (for loading from the local namespace, inside a function), 'LOAD_GLOBAL' (for loading from the global or builtin namespace, skipping the local namespace), and 'LOAD_NAME' (for occurances where it's truly not known where a variable comes from, and both local and global namespaces have to be searched.) The latter is necessary because Python has a few operations that can modify a local namespace at runtime (such as 'from module import *', and 'exec'.) Since Python 2.1, Python has 'nested scopes', which makes code blocks defined inside other code blocks slightly special: the 'parent' scope is also searched for any variable that isn't local to the nested scope. Most of this, but not all, is done at compiletime, IIRC. (Jeremy can correct me if I'm wrong :) I tried to make the distinction between a 'function' and a 'code block', since the VM really deals with the second, not the first. I suspect that the current Python VM's mechanics to deal with nested scopes could be generalized so that Perl's lexical closures could fit as well (I assume that won't change in perl6 :). Then again, Perl already knows where a variable comes from, so the VM will probably want to put that to good use... The main reason Python does the name-searching at runtime is that a global namespace can be altered from outside the currently-executing codeblock (threads, or other modules, can do 'module.range = my_fake_range', and suddenly the range() function does something completely different throughout the entire module.) A good overview of how the VM uses the API can be obtained by looking at Python/ceval.c, and looking for "switch (opcode)". You'll end up at the top of the main bytecode switch. Most of the opcode names are pretty descriptive, but if you want a better overview, take a look at the 'dis' module's documentation. (www.python.org is still down, but you can use one of the mirrors: http://python.mirrors.netnumina.com/doc/current/lib/module-dis.html http://python.mirrors.netnumina.com/doc/current/lib/bytecodes.html And an easy way to see how a Python operation is actually implemented is using the 'dis' module interactively:
3 SET_LINENO 2 6 SETUP_LOOP 44 (to 53) >> 9 SET_LINENO 2 12 LOAD_FAST 0 (x) 15 JUMP_IF_FALSE 33 (to 51) 18 POP_TOP 19 SET_LINENO 3 22 LOAD_CONST 1 ('Hello ') 25 LOAD_CONST 2 ('Perl ') 28 BINARY_ADD 29 LOAD_CONST 3 ('World!') 32 BINARY_ADD 33 PRINT_ITEM 34 PRINT_NEWLINE 35 SET_LINENO 4 38 LOAD_FAST 0 (x) 41 LOAD_CONST 4 (1) 44 INPLACE_SUBTRACT 45 STORE_FAST 0 (x) 48 JUMP_ABSOLUTE 9 >> 51 POP_TOP 52 POP_BLOCK >> 53 LOAD_CONST 0 (None) 56 RETURN_VALUE Keep in mind that this is stack-based, and all operations manipulate the stack. (for instance, LOAD_FAST pushes the variable onto the stack, and BINARY_ADD pops two values, adds them, and pushes the result.) -- Thomas Wouters <thomas@xs4all.net> Hi! I'm a .signature virus! copy me into your .signature file to help me spread!

On Wed, Aug 01, 2001 at 03:36:07PM +0200, Thomas Wouters wrote:
This looks rather like a Perl SV (scalar value): struct STRUCT_SV { /* struct sv { */ void* sv_any; /* pointer to something */ U32 sv_refcnt; /* how many references to us */ U32 sv_flags; /* what we are */ };
We use double indirection so that we can switch between types very quickly; the sv_any pointer points to another structure or an integer/float.
The struct _typeobject pointer is a pointer to a Type object, which also acts as a vtable.
That's sounding more like how we plan to do things for Perl 6.
% ./perl -Dts -e 'print "Hello Python people!\n"' => (-e:1) null => (-e:1) const(PV("Hello Python people!\12"\0)) => PV("Hello Python people!\12"\0) (-e:1) stringify [ That was constant folding, BTW. - SC] EXECUTING... => (-e:0) enter => (-e:0) nextstate => (-e:1) pushmark => * (-e:1) const(PV("Hello Python people!\12"\0)) => * PV("Hello Python people!\12"\0) (-e:1) print Hello Python people! => SV_YES (-e:1) leave perl -MO=Bytecode,-S -e 'print "Hello Python people!\n"' will give you a complete disassembly of the bytecode, but that's the version that can be serialised to disk, so it's a lot more verbose than it need be - it sets up all the opcodes, their flags, their sequence numbers, their connections to other nodes in the opcode tree and so on. If you really want to get stuck in, read the internals tutorial at http://www.netthink.co.uk/downloads/ Simon

At 03:36 PM 8/1/2001 +0200, Thomas Wouters wrote:
Ah, I see. (and thanks heaps) Well, I just rejigged my thinking a bit on what we need to export, then. My take's been both more agressively black-box about things, and much less OO. Too much C on the brain, I expect. Dan --------------------------------------"it's like this"------------------- Dan Sugalski even samurai dan@sidhe.org have teddy bears and even teddy bears get drunk

Andrew Kuchling wrote:
Welcome to Perl. :) I don't really understand it but here are references that might help: http://aspn.activestate.com/ASPN/Mail/Message/638953 http://aspn.activestate.com/ASPN/Mail/Message/639000 http://aspn.activestate.com/ASPN/Mail/Message/639048 -- Take a recipe. Leave a recipe. Python Cookbook! http://www.ActiveState.com/pythoncookbook

Thomas Wouters writes:
Perl's REs now do Unicode. Perl 6's REs will do named groups.
And I won't even start with Perl's more archaic features, that change the whole working of the interpreter.
Those are going away. Perl people hate them as much as you do--the only time they're used now is to make deliberately hideous code, and hardly anyone will seriously lament the passing of that ability. No more "change the starting position for subscripts", no more "change all RE matches globally", and so on. Nat

On Mon, Jul 30, 2001 at 08:12:04PM -0700, Nathan Torkington wrote:
Yeah, I thought as much, which is why I wasn't going to start on them :)
I don't really hate the features, I just don't use them, and wouldn't want them in Python :-) I do actually program Perl, and will do a lot more of it in the next couple of months at least (I switched projects at work, to one that will entail Perl programming roughly 80% of the time) -- I just like Python a lot more. Your comments do lead me to ask this question, though (and forgive me if it comes over as the arrogant ranting of a Python bigot; it's definately not intended as such, even though I only have a Python-implementors point of view.) What's going to be the difference between Perl6 and Python ? The variable typing-naming ($var, %var, etc) I presume, and the curly bracket vs. indentation blocking issue. Regex-literals, 'unless', the '<expression> if/unless/while <boolean exp>' shortcut, I guess ? Those are basically all parser/compiler issues, so shouldn't be a real problem. The transmorphic typing is trickier, as is taint mode and Perl's scoping rules.... Though the latter could be done if we refactor the namespace-creation that is currently done implicitly on function-creation, and allow it to be done explicitly. The same goes for the variable-filling-assignment (which is quite different from the name-binding assignment Python has.) I don't really doubt that Perl and Python could use the same VM.... I'm not entirely certain howmuch of the shared VM the two implementations would actually be using. Is it worth it if the overlap is a mere, say, 25% ? (I think it's more, but it depends entirely on howmuch different Perl6 is from Perl5, and howmuch Python is willing to change.... Lurkers here know I'm agressively against gratuitous breakage :) -- Thomas Wouters <thomas@xs4all.net> Hi! I'm a .signature virus! copy me into your .signature file to help me spread!

One of the things I picked up from the Perl conference is that Perl users *seem* (to me) to have a higher tolerance for code breakage than Python users. (and Python users have a higher tolerance than (let's say) Java users) Even if we put aside Perl 6, Perlers talk pretty glibly about ripping little used features out in Perl 5.8.0 and Perl 5.10 and so forth. e.g. Damian said that Autoload is going away (or pseudo hashes or something like that). Whether or not he was right, nobody in the room threw tomatoes as I'm sure they would if Guido tried to kill __getattr__. Admittedly, I never know when I hear stuff like "tr///CU is dead" or "package; is dead" whether each was a feature that has been in for three years or was added to an experimental release and removed from the next experimental release. I'm not criticizing the Perl community. Acceptance of change is a good thing! But I think they should know how conservative the Python world is. Last week there were storm troopers heading for Guidos house when he announced that the division operator is going to change its behaviour two or three years. That means it would take a major PR effort to convince the Python community that even minor language changes would be worth the benefit of sharing a VM. -- Take a recipe. Leave a recipe. Python Cookbook! http://www.ActiveState.com/pythoncookbook

Paul Prescod writes:
(off-topic, but he started it!) That's a really interesting comment. In many ways it's probably true, although make no mistake about it--we take it in the neck for every language change we make! The features we're talking about removing are either deprecated or experimental. With that said, we've probably been less rigorous than we should be about segregating experimental features from blessed ones. I suspect that because experimental features are the only way to do something, they still get used no matter how much we caution that they may go away. I've been eyeing your 'import from the future' system of getting experimental features. I'll be interested to see how this holds up. Perl and Python believe in active maintenance and development with many small incremental releases (unlike Java with occasional bursty releases, and C++ and C which are to all intents and purposes static). Changes to the language cause problems we share, and the other mainstream languages don't provide us with any direction on how to approach the problem. We in the Perl camp have bandied around the idea of "use perl 5.004" as a way to get all the behaviour of the 5.004 release. That's a noble idea, but forces us to keep a lot of crap around when one of the goals of active development is to be able to shed obstructive missteps in language evolution. Nat

On Tue, Jul 31, 2001 at 08:26:47PM -0600, Nathan Torkington wrote:
I've been eyeing your 'import from the future' system of getting experimental features. I'll be interested to see how this holds up.
It's really not that different form Perl's "use". They're both scanned for by the parser/tokenizer, they're both used to change the semantics of an operation, and they also double as a normal 'get and use a package' operation (which is, in fact, the primary task of 'import' and 'use' both, right ? :) The main difference is that changing the behaviour of the Python interpreter is a lot less common than changing the behaviour of the Perl one, and Guido wants to keep it that way. The purpously 'magic' spelling of the future-import statement is part of that. -- Thomas Wouters <thomas@xs4all.net> Hi! I'm a .signature virus! copy me into your .signature file to help me spread!

Nathan Torkington wrote:
Actually, Python doesn't usually have experimental features. "import from future" is about introducing new features that will break code "early". On the one hand, some people want the feature as soon as possible. On the other, people are afraid of the code breakage (e.g. adding a keyword in Python breaks code) and need time to adjust. The future construct is the compromise. Python has no real mechanism for experimental features other than documentation, patches and so forth. I can only imagine one or two experimental features in the history of the language, though.
Yes, Guido has rejected that idea for Python as too much effort. Imagine the regression testing problems alone! -- Take a recipe. Leave a recipe. Python Cookbook! http://www.ActiveState.com/pythoncookbook

Since python does so much by name (I think locals are the only place where name lookup is compiled away), dictionary lookup is pretty fundamental to the runtime, and is used by a number of opcodes. No reason that dictionary lookup couldn't be part of the common runtime, or else tucked behind an abstract interface common to Python and Perl lookups, but it's something to keep in mind if the VM is going to operate on a similar level to the current one. But then, I've always thought that one of the problems with trying to optimize Python was that the VM was too high level. If some sort of Forth-like extensible threaded code were used, you could build the current opcodes from lower level primitives. Re: stack vs. register machines: Some Forth implementations cache some of the top of stack in registers, but the more you try to cache, the hairier it gets. ( But you can figure out the bookkeeping once and automatically generate the code variations. ) You might take a look at Anton Ertl's VMGEN: <http://www.complang.tuwien.ac.at/anton/vmgen/> | Vmgen generates much of the code for efficient virtual machine (VM) | interpreters from simple descriptions of the VM instructions. [ One of the nice/useful features of the the Forth VM is the PFA/CFA pairing: PFA is "parameter field address" and points to the code ( VM or native ) to be executed. CFA is "code field address" and points to the code to interpret what's in the parameter field. For threaded code, it points to the threaded code interpreter; for native code, it points to the PFA -- i.e. native code is 'self interpreting' . BUILDS/DOES in Forth creates a data type (BUILDS) and defines code to addess the data type (DOES) that is pointed to by the CFA -- an early but very primitive object-orientation, but with only one method (later Forth's added QUADS and other methods to have separate ACCESS/UPDATE (read/write) methods. ] Re: Other VM implementations: I'm not very familiar with the internals of Squeak, but I suspect that it's worth looking at. They are, in any case, interested in some of the same sort of things. ( There was a recent thread about MIT's StarLogo -- which was originally written for the Mac using (I think) Lisp, and then a portable version was done using Java, but they were disappointed in performance, and I think they are looking at using Squeak now. ) Scheme48 is probably considered the best portable byte-code Scheme implementation. ( Don't know anything about it's internals myself ) A lot of other people who have tried using the Java VM for other languages have had complaints about various things that are difficult or impossible. ( Scheme folks couldn't have full call/cc, and there were two different attempts to add generics to Java -- one involved adding special bytecode support, and the other (Pizza -- now GJ -- Generic Java) tried to stick with a standard Java VM. ) -- Steve majewski

On Mon, 30 Jul 2001, "Steven D. Majewski" <sdm7g@Virginia.EDU> wrote:
Scheme48 is probably considered the best portable byte-code Scheme implementation. ( Don't know anything about it's internals myself )
Last I heard (admittedly, >1 yr. ago), it didn't support 64 bit architectures. -- gpg --keyserver keyserver.pgp.com --recv-keys 46D01BD6 54C4E1FE Secure (inaccessible): 4BD1 7705 EEC0 260A 7F21 4817 C7FC A636 46D0 1BD6 Insecure (accessible): C5A5 A8FA CA39 AB03 10B8 F116 1713 1BCF 54C4 E1FE Learn Python! http://www.ibiblio.org/obp/thinkCSpy

"Steven D. Majewski" <sdm7g@Virginia.EDU>:
But then, I've always thought that one of the problems with trying to optimize Python was that the VM was too high level.
No, the problem is that Python is just too darn dynamic! This is a feature of the language, not just the VM. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+

I was thinking a little about a Python/Perl VM merge. One problem I imagine would be difficult to reconcile is the subtle difference in semantics of various basic types. Consider the various bits of Python's (proposed) number system that Perl might not have (or want): rationals, automatic promotion from machine ints to longs, complex numbers. These may not work well with Perl's semantics. What about exceptions? Do Python and Perl have similar notions of what exceptional conditions exist? Skip

Actually, this may not be as big a deal as I thought before. The PVM doesn't have a lot of knowledge about types built into its instruction set. It knows a bit about classes, lists, dicts, but not e.g. about ints and strings. The opcodes are mostly very abstract: BINARY_ADD etc. --Guido van Rossum (home page: http://www.python.org/~guido/)

Guido> The PVM doesn't have a lot of knowledge about types built into Guido> its instruction set.... The opcodes are mostly very abstract: Guido> BINARY_ADD etc. Yeah, but the runtime behind the virtual machine knows a hell of a lot about the types. A stream of opcodes doesn't mean anything without the semantics of the functions the interpreter loop calls to do its work. I thought the aim of Eric's Parrot idea was that Perl and Python might be able to share a virtual machine. If both can generate something like today's BINARY_ADD opcode, the underlying types of both Python and Perl better have the same semantics. Skip

Skip Montanaro wrote:
I don't think that needs to be true _in toto_. In other words, some opcodes can be used by both languages, some can be language-specific. The implementation of the VM for a given opcode can be shared per language, or even just partially shared. BINARY_ADD can do the same thing in most languages for 'native' types, and defer to per-language codepaths for objects, for example. One problem with a hybrid approach might be that optimizations become really hard to do if you can't assume much about the semantics, or if you can only assume the union of the various semantics. But the idea is intriguing anyway =). --david

Yeah, but the runtime could offer a choice of data types -- for Python code the constants table would contain Python ints and strings etc., for Perl code it would contain Perl string-number objects. Maybe. --Guido van Rossum (home page: http://www.python.org/~guido/)

Guido> Yeah, but the runtime could offer a choice of data types -- for Guido> Python code the constants table would contain Python ints and Guido> strings etc., for Perl code it would contain Perl string-number Guido> objects. Maybe. So I could give a code object generated by the Python compiler to the Perl runtime and get different results than if it was executed by the Python environment? Perhaps it's time for Eric to chime in again and tell us what he really has in mind. I can't see the utility in having the same set of opcodes for the two languages if the semantics of running them under either environment aren't going to be the same. It seems like it would artificially constrain people working on the internals of both languages. Skip

Skip Montanaro <skip@pobox.com>:
No, I don't think that's what Guido is saying. He and I are both imagining a *single* runtime, but with some type-specific opcodes that are generated only by Perl and some only generated by Python.
You're right. What I have in mind starts with a common opcode interpreter, perhaps based on the Python VM but with extended opcodes where Perl type semantics don't match, and a common callout mechanism to C-level runtime libraries linked to the opcode interpreter. In the conservative version of this vision, Perl and Python have different runtimes dynamically linked to an instance of the same opcode interpreter. Memory allocation/GC and scheduling/threading are handled inside the opcode interpreter but the OS and environment binding is (mostly) in the libraries. Things Python would bring to this party: our serious-cool GC, our C extension/embedding system (*much* nicer than XS). Things Perl would bring: blazingly fast regexps, taint, flexitypes, references. In the radical version, the Perl and Python runtimes merge and the differences in semantics are implemented by compiling different wrapper sequences of opcodes around the library callouts. At this point we're doing something competitive with Microsoft's CLR. My proposed work plan is: 1. Separate the Python VM from the Python compiler. Initially it's OK if they still communicate by hard linkage but that will change later. 2. Build the Parrot VM out from the Python VM by adding the minimum number of Perliferous opcodes. 3. Start building the Perl runtime on top of that, re-using as much of the Python runtime as possible to save effort. -- <a href="http://www.tuxedo.org/~esr/">Eric S. Raymond</a> Every election is a sort of advance auction sale of stolen goods. -- H.L. Mencken

[Eric, could you forward this to python-dev if it doesn't show of its own accord? I'm not yet subscribed, so I don't know if it'll make it] I should start with an apology for not being on python-dev when this started. Do please Cc me on anything, as I've not gotten on yet. (My subscription's caught in the mail, I guess... :) At 04:14 AM 7/31/2001 -0400, Eric S. Raymond wrote:
Odds are there won't even be a different set of opcodes. (Barring the possibility of the optimizer being able to *know* that an operation is guaranteed to be integer or float, and thus using special-purpose opcodes. And that's really an optimization, not a set of language-specific opcodes) The behaviour of data is governed by the data itself, so Python variables would have Python vtables attached to them guaranteeing Python behaviour, while perl ones would have perl vtables guaranteeing perl behaviour. This was covered, more or less, by the chunks of the internals talk I didn't get to. Slides, for the interested, are at http://dev.perl.org/perl6/talks/. I'm not sure if there's enough info on the slides themselves to be clear--they were written to be talked around.
I've snipped the rest here. I don't think Parrot will be built off the Python interpreter. This isn't out of any NIH feelings or anything--I'm obligated to make it work for Perl, as that's the primary point. If we can make Python a primary point too that's keen, and something I *want*, but I do need to keep focused on perl. Having said that, what I'm doing is stepping back from perl and trying, wherever possible, to make the runtime generic. If there's no reason to be perl specific I'm not, and so far that's not been a problem. (It actually makes life easier in a lot of ways, since we can then delegate the decision on how things are done to the variables involved, providing a default set of behaviours which the parser will end up determining anyway) On some things I think I'm being a bit more vicious than, say, Python is by default. (For example, if extension code wants to hold on to a variable across a GC boundary it had darned well better register that fact with the interpreter, or it's going to find itself with trash) I'm not sure about the extension mechanism in general--I've not had a chance to look too closely at what Python does now, but I don't doubt that, at the C level, the differences between the languages will be pretty trivial and easily abstractable. Seeing what you folks have is on the list 'o things to do--I may well steal from it wholesale. :) I expect there's a bunch of stuff I'm missing here, so if anyone wants to peg me with questions, go for it. (Cc me if they're going to the dev list please, at least until I'm sure I'm on) I really would like to see Parrot as a viable back end for Python--I think the joint development resources we could muster (possibly with the Ruby folks as well) could get us a VM for dynamically typed languages to rival the JVM/.NET for statically typed ones. Dan --------------------------------------"it's like this"------------------- Dan Sugalski even samurai dan@sidhe.org have teddy bears and even teddy bears get drunk

Hm, the Python bytecode interpreter doesn't do the memory allocation/GC, and only does the scheduling/threading indirectly (by releasing and immediately reacquiring the global interpreter lock).
I'm not sure I want references, and I don't know what flexitypes are. --Guido van Rossum (home page: http://www.python.org/~guido/)

At 07:15 PM 7/31/2001 -0400, Guido van Rossum wrote:
The Parrot interpreter does (or will at least, as there's not much code behind it yet) handle GC and intercept memory allocation, at least if you want to use managed memory. (Though that's only of interest to people who are writing C/C++/Fortran/Whatever extensions to the interpreter) GC & object destruction are going to be dealt with separately by the interpreter, FWIW. Which might not be much. As for threading, well, that's where things get interesting. Perl's tried it two ways (multiple threads in the same interpreter, and one thread per interpreter, with cloned interpreters) both of which aren't very good. And the global lock thing's not that keen either. The plan for parrot is to allow multiple interpreters to run simultaneously, with shared data protected on access time. (And *only* shared data protected, so there's no penalty to access non-shared data) Along with a scheme to allow only partial (or no) visibility of the spawning thread's variables. We plan on basing the secure sandbox scheme around this as well.
You certainly don't have to have them. The parser doesn't have to necessarily expose all the features of the interpreter. What you'd expose to the programmer would depend on the language grammar--if you can't write valid Python code to take references, well, you don't get them. :)
and I don't know what flexitypes are.
I presume this is perl's "I'll make it any type I darned well need" basic scalar, or possibly it's auto-resizing arrays and hashes. Neither of which you have to have either. Since that behaviour's a function of the type of a variable, we'd just not give python variables perl variable types. (Or the vtables that go with them) Now, what happens when a Python routine is passed a perl variable is an interesting question. Personally I'd presume that it Did The Right Thing, but I'm not 100% sure what that is. Probably maintain its core behavior. (So if I passed in a Magic Morphing Perl Scalar to a Python routine that needed an integer value, it'd get one, but if I tried to use a Python integer in string context in a perl sub, I'd get yelled at. (Assuming that's the correct behaviour, of course. If not, well, it'd do something else)) Dan --------------------------------------"it's like this"------------------- Dan Sugalski even samurai dan@sidhe.org have teddy bears and even teddy bears get drunk

Dan Sugalski wrote:
What is the downside of the global lock on the average single processor machine? I tend to think that the "default" threading model should allow simple and easy, everything-shared multi-threading on ordinary machines. Having a multi-processor-friendly advanced mode is a great extension for the wizards. -- Take a recipe. Leave a recipe. Python Cookbook! http://www.ActiveState.com/pythoncookbook

At 04:42 PM 7/31/2001 -0700, Paul Prescod wrote:
If you hold the lock during an I/O operation, you'll lose time you could have otherwise used. Getting and releasing a global lock frequently also costs performance you might otherwise have used in other places. Mutex releases require memory coherency, which will force your CPU to flush any pending writes that might be hanging about, which will tend to drop it's efficiency, especially on heavily out-of-order machines like the Alpha. Also, that is a zillion and a half mutex aquisition and releases, most of which you probably have no need of. Even cutting out, say 5-10% of them will be noticeable. (I do work on SMP machines as a rule, so I am a little biased against things that single-thread me when I don't need it--what's the point of 500% idle time?) Dan --------------------------------------"it's like this"------------------- Dan Sugalski even samurai dan@sidhe.org have teddy bears and even teddy bears get drunk

To bring in yet someone else to this discussion, Mark Hammond and I were chatting with Brendan Eich (of JavaScript and Mozilla fame) last week about interpreter locks and the like, and Brendan mentioned that he had done a lot of possibly reusable work in optimizing the various kinds of thread lock strategies in JavaScript, as JS is used a lot server-side. Specifically he was talking about some 'multi-level' locking stuff which sounded impressive to me at least. I doubt if Brendan has time to dig into either of Perl or Python's internals, but I'm sure he'll provide general pointers if folks ask nicely. =) Brendan, do you care to give an executive overview of the locking strategies used in JS? --david ascher PS: Brendan: If you want to know the context of this discussion, see: http://aspn.activestate.com/ASPN/Mail/Browse/Threaded/python-dev

BTW, for a VM flexible enough to deal with Python and Perl, JavaScript should be a piece of cake. (JavaScript is much closer to Python than to Perl in its semantic model, for the most part.) --Guido van Rossum (home page: http://www.python.org/~guido/)

Dan Sugalski wrote:
"Doctor it hurts when I do that." :) We just don't hold the lock during I/O or other computations that do not affect Python interpreter data structures. -- Take a recipe. Leave a recipe. Python Cookbook! http://www.ActiveState.com/pythoncookbook

Dan Sugalski wrote:
If you hold the lock during an I/O operation, you'll lose time you could have otherwise used.
Python extensions can (and do, for all standard I/O operations) release the Global Interpreter Lock. For more information, check out http://starship.python.net/crew/aahz/ -- --- Aahz (@pobox.com) Hugs and backrubs -- I break Rule 6 <*> http://www.rahul.net/aahz/ Androgynous poly kinky vanilla queer het Pythonista I don't really mind a person having the last whine, but I do mind someone else having the last self-righteous whine.

[Paul Prescod]
[Dan Sugalski]
Python doesn't actually suffer from either of these problems: while there's a pair of acquire/release-global-lock macros around potentially blocking I/O calls in Python's runtime (ditto sleep(), etc), no mutex is actually allocated before *somebody* calls PyEval_InitThreads: void PyEval_InitThreads(void) { if (interpreter_lock) return; _PyThread_Started = 1; interpreter_lock = PyThread_allocate_lock(); PyThread_acquire_lock(interpreter_lock, 1); main_thread = PyThread_get_thread_ident(); } In most uses of Python, no thread other than the main thread ever gets created, that routine never gets called, interpreter_lock remains NULL, and all the global-lock acquire/release code reduces to a cheap test against NULL. However, Python calls the platform's thread-safe libraries regardless, and *that* can be a huge speed hit. A minor example is that system malloc() is more expensive in Microsoft's thread-safe version of libc. A monster example is speed of line-at-a-time input: we only recently discovered that Python's getc()-in-a-loop was killing us on many platforms because the platform threadsafe library implementation locked and unlocked the stream for each character. Worming around that brought our input speed much closer to Perl's (up to 50x faster on Tru64 Unix). It's still slower on most boxes, though, because we're still threadsafe, but, last I looked, Perl's line-at-a-time input tricks mucked with stdio structs directly without benefit of exclusion (and are not threadsafe).
Greg Stein is the fellow to talk with about about "free threading" of Python. He had that at least mostly working several years ago, but it was a major project, that patch is way out of date now, and Python is much more elegant now. Oops! I didn't mean "elegant", I meant "bigger" <wink>.

At 01:23 AM 8/1/2001 -0400, Tim Peters wrote:
Yeah, I figured you didn't initialize or aquire the mutex until it was actually needed. One of the nice side-benefits of the global opcode-aquired lock.
Everywhere else too, I'd bet. I've been considering thread-specific memory pools because of this. (Well, this is one reason, at least)
Ouch, I'd bet that hurts. Has anyone timed the difference between making lots of getc calls and making a few larger reads and managing the buffers internally? I can see it going either way, and another data point would be useful to have.
Worming around that brought our input speed much closer to Perl's (up to 50x faster on Tru64 Unix).
FWIW, at least on Tru64 and VMS, there are a number of faster thread calls if you don't mind bypassing some library error checking. (Which is OK if you're guaranteed to have correct parameters) I can dig them up if you like. A tweak of pthread_get_specific made a depressingly large performance boost for me on my VMS box. (I did say perl's threading models weren't that good... :) Not worth it if you don't make many calls, though.
Yep, perl's pthread-based threading model doesn't guarantee threadsafe I/O on many platforms. The alternative threading model doesn't use that codepath. (It's currently primarily windows-based, and we don't do the buffer lookbehind stuff there)
:) I'm as much (or more, which is generally an odd thing, but threads are almost inherently odd) interested in how things look at the programmer level and what guarantees are made as how things look under the hood. (It's all just a SMOP, right?) Dan --------------------------------------"it's like this"------------------- Dan Sugalski even samurai dan@sidhe.org have teddy bears and even teddy bears get drunk

[Dan Sugalski, on Wednesday, August 01, 2001 2:20 AM]
There's been lots of this in Python-Dev, like in this thread: http://aspn.activestate.com/ASPN/Mail/Message/600485 I'll quote the high-order bit: My line-at-a-time test case used (rounding to nearest whole integers) 30 seconds in Python and 6 in Perl. The result of testing many changes to Python's implementation was that the excess 24 seconds broke down like so: 17 spent inside internal MS threadsafe getc() lock/unlock routines 5 uncertain, but evidence suggests much of it due to MS malloc/realloc (Perl does its own memory mgmt) 2 for not copying directly out of the platform FILE* implementation struct in a highly optimized loop (like Perl does) My last checkin to fileobject.c reclaimed 17 seconds on Win98SE while remaining threadsafe, via a combination of locking per line instead of per character, and invoking realloc much less often (only for lines exceeding 200 chars). Note that thread overhead is overwhelmingly the biggest hangup. Python has two threadsafe input tricks now: 1. On platforms that have flockfile(), funlockfile(), and getc_unlocked(), the last is used in a loop bracketed by the first two. 2. At least on Windows, which doesn't have those, we use the platform fgets() in an excruciating way, tricking it into letting us read lines with embedded null bytes. Oddly enough, in the timing reports I saw, approach #1 was never faster than approach #2, and on at least one platform (Tru64, IIRC) was slower. Of course fgets() is a primitive in std C because they *wanted* to make it possible for vendors to optimize it (in the ways Perl does), but it appears very few vendors do optimize it. On Windows it's the same old getc()-in-a-loop, but they lock/unlock the stream only once per fgets call (using internal stream functions that aren't exposed). The "2 seconds for not copying directly ... like Perl does" I reported above came from hacking together a thread-unsafe line input routine that used the same FILE* tricks Perl uses. That is, thread-unsafe getc-in-a-loop was 2 seconds slower than using thread-unsafe FILE* tricks. That's significant in absolute terms, but was lost in the noise compared to the other stuff we were fighting.

I don't have context for this thread anymore, so perhaps this is already what's being discussed: Perhaps a less ambitious (than Parrot) but still very useful project would be a portable stdio-like library that has the features dynamic languages need? This could be based upon a few low-level calls like read() and write() on Unix. These tend to exist in most platforms C libraries (even on Windows) because they're so darned useful. For full "portability" it should be possible to implement these yourself if the platform doesn't have them (or if they are not fast enough!). --Guido van Rossum (home page: http://www.python.org/~guido/)

On Sun, Aug 12, 2001 at 12:11:56AM -0400, Guido van Rossum wrote:
I hope you do not think that I'm blowing Perl's trumpet again, but I see one of my roles here as sharing knowledge about things that Python has that Perl needs and vice versa. Perl 5.7.x has such a library, and it's my expectation (another sideways glance at Dan) that if we did Parrot, it would include this library. PerlIO attempts to be a very portable stdio (it's running on everything that Perl runs on, AFAIK) and it supports "line disciplines" in the sfio sense: you can stuff processing modules between the file and the filehandle. This allows us to do anything between altering buffering semantics, and decompressing and decrypting files on the fly. It'll need to be somewhat rewritten for Perl 6, at which point I'm hoping it'll be done in such a way that it'll make it easier to extract from the Perl mess around it and be made available to other projects. -- Putting heated bricks close to the news.admin.net-abuse.* groups. -- Megahal (trained on asr), 1998-11-06

Great!
I hope that even if the Parrot goal is to support all languages, PerlIO can be made a separate and independent project, not relying on Perl internals. (And I'd change the name, but maybe that's asking too much. :-) --Guido van Rossum (home page: http://www.python.org/~guido/)

On Sun, 12 Aug 2001, Guido van Rossum wrote:
Two of the other design requirements are good async I/O support and good event handling support. (So there's something to handle the completed async I/Os) Rumors that we're just filing the serial numbers off the VMS I/O system are, of course, wildly exaggerated. ;-)
Nah, a name change would certainly be in order--we'd have to call it the ParrotIO system. :) Making the core bits modular is one of the goals, for no other reason than to make it easy to have someone responsible for its maintenance and development that doesn't need to be intimately familiar with the rest of the interpreter system. And so it'll still function properly when we compile down to native code and don't actually *have* an interpreter handy. It ought to be an interesting project. Suggestions for features are, of course, always welcome. Dan

Dan Sugalski wrote:
Would it make sense to poll both communities for people interested in helping with this effort and maybe making it a standalone project (perhaps on SourceForge?)? In fact, there are many bits of the C library that Python people would like to replace or improve but we don't have the resources alone. Maybe if all of the scripting language groups got together we would be able to do it. I'm not talking about the larger project of sharing bytecodes and so forth, but I could see that we probably have very similar needs with respect to files, memory allocation, networking, event loops, process handling and maybe threads. And the other scripting languages are probably not too far off either. -- Take a recipe. Leave a recipe. Python Cookbook! http://www.ActiveState.com/pythoncookbook

Paul Prescod <paulp@ActiveState.com>:
I think this is a good idea in itself. And would also be a good pilot project for the Parrot unification. -- <a href="http://www.tuxedo.org/~esr/">Eric S. Raymond</a> The direct use of physical force is so poor a solution to the problem of limited resources that it is commonly employed only by small children and great nations. -- David Friedman

Can I make a request? Can we *not* crosspost between language-dev and python-dev? The whole point of language-dev is to keep the traffic *out* of the language-specific lists. Please trim your "to" lines. --Guido van Rossum (home page: http://www.python.org/~guido/)

On Sun, Aug 12, 2001 at 04:41:51PM -0700, Paul Prescod wrote:
Yes. But you'll have to talk to Nick (<nick@ing-simmons.net>) if you want to play with the current code, since it's his baby.
In fact, there are many bits of the C library that Python people would like to replace or improve but we don't have the resources alone.
*nod*. Perl reimplements far too much of C for my liking. (aton, sprintf, malloc [1] and friends.) I keep telling 'em to use glib. [1] ... but then any C program over a given size reimplements malloc ... -- The best book on programming for the layman is "Alice in Wonderland"; but that's because it's the best book on anything for the layman. - Alan Perlis

On Sun, 12 Aug 2001, Paul Prescod wrote:
I know that there are some PHP developers who are also thinking along the same lines -- myself for one. :) Also, another area this might reach is into the extension space, we're all developing the same extension for each of the different languages, it would be great to find a way to pool the resources of all the extension developers to create one extension together (ie, a singular XSLT extension, used in Perl, Python, PHP, etc.) -Sterling

On Mon, 13 Aug 2001, Sterling Hughes wrote:
By introducing another layer of abstraction we will most probably decrease the performance of all languages[1]. That is, since the universal interface to, given your example, the Sablotron XSLT library is the API and Perl, Python and PHP have internals different enough that there would have to be a certain amount of glue and translation work between the structures of the extension layer and the internals of the interpretter. Building extensions to these languages involves a lot of hack work I agree. The reason for this is that it is merely connecting the wires between an API and the language of interest. Anything more than that and you'll be damaging the performance of Perl, Python, PHP... [1] The situation where you could perhaps increase performance is the converse of what is being suggested. Presumably, if it is in the interest of language developers to have a unified extension layer, then the point of different which language developers want to maintain with with other languages is the syntax/grammar of the language. As I see it, along this line of thinking it makes sense to look at the feasibility of a language system (internals and extension layer) which handles plugable scripting engines. As such, the majority of work which goes into language development can be concentrated while allowing, to the users, automony in their choice of language.
-Sterling
Gavin

It should still be possible to come up with some guidelines for writing a single extension that supports multiple scripting languages. Especially for something like Sablotron which isn't that complex. You feed it some XML and some XSL and it spits back a result. Most of the work in writing the extension is figuring out how the user-visible API should look and then translating that API along with its function/method calls and argument type mangling to whatever the thing that is being glued is expecting. Note that I am not talking about runtime binary compatibility here. I am talking about source compatibility where potentially big chunks of code would be very different across the different languages. In an approach like this I don't see a performance issue. It is more of an education issue actually. And with a bit of effort I bet we could come up with enough common ground, or at least some nifty macro tricks, where people wouldn't feel overwhelmed by the task of supporting multiple scripting languages when they sat down to write some new backend library. Perhaps it would be a good experiment to take some really simple thing and write Perl, Python, PHP, Ruby, and Tcl extensions for it and compare notes. See where the overlap is, see where the biggest divergence is, and figure out if some of that divergence could be eliminated with some clever tricks.
Sure, but now you are back on the holy grail and CLR stuff again. Baby steps... -Rasmus

On Sun, 12 Aug 2001, Rasmus Lerdorf wrote:
Well -- Sablotron is more of an annoyance in this regard than anything -- if you want to support SAX, Scheme, error handling, log handling and such, you need to register a million callbacks and then have each of these callbacks either do something useful, or call some user space function. puhhhh. The PHP-XSLT extension, all in all is 2525 lines of code, Makefiles, and explanations, 2525 more than I would've liked to write (http://cvs.php.net/cvs.php/php4/ext/xslt)... Using PHP-Sablotron as an example is something where having a couple of minds is better... The first version of the extension (http://cvs.php.net/cvs.php/php4/ext/sablot), left, uhhmm, something to be desired -- the api was a bit hard to understand, and it made code maintainability within the extension something close to hell. Perhaps if there were a couple more minds working on the extension, some of the mistakes of the original version might not have been present (perhaps if I had slept the weekend that I wrote it, that might have happened as well, who knows? :) Anyway, just counting now -- 4530 lines of "stuff" :) have been written on the PHP end of things for bring sablotron support in (actually, the newer XSLT extension will also support libxslt and Xalan, so some of its bloat for those extra libraries, but..) I imagine that on the Perl and Python ends at least 1,000 lines of code have been written on each side. Wouldn't it be spiffy if this only had to be done once? Hell, if there's a singular extension api, as you say, vendors may just write to that spec, and we don't have to worry about writing it at all!
I think this would be interesting at least -- I'll write the PHP extension (Hell, I already have, http://cvs.php.net/cvs.php/php4/ext/zip, it don't get easier than that :) -Sterling

On 12 August 2001, Paul Prescod said:
Isn't a lot of that addressed by glib, the C library used by GTK+ (and therefore Gnome) applications for all those things that C programs tend to do that aren't GUI-related? And speaking of I/O libraries, whatever happened to sfio? I remember struggling with it and PerlIO back around Perl 5.003/5.004 in a vain attempt to get FastCGI working. sfio seemed at the time like a good idea just didn't catch on, probably because stdio is "good enough" for most people. (But then, "most people" don't implement Perl or Python.) (I also remember that sfio's version number is the year of release -- so much for "release early, release often". It also had a rather idiosyncratic build system -- autoconf? what's that?) Greg -- Greg Ward - Unix bigot gward@python.net http://starship.python.net/~gward/ I'd rather have a bottle in front of me than have to have a frontal lobotomy.

Dan Sugalski wrote:
This is where calling what Python uses "variables" gets you into real problems. Many people think that the terms for "variable" and "reference" should be "name" and "binding"; it makes things far less confusing. See, Python objects are *global*. Period. No ifs, ands, or buts. Any object can be shared at any time. More than that, most of the time Python threads are bound to class instances, so causing another thread to get a binding to an object is as simple as setting an attribute of that thread's instance. And, as Paul pointed out, the global interpreter lock only kills you if you're running SMP in the first place. -- --- Aahz (@pobox.com) Hugs and backrubs -- I break Rule 6 <*> http://www.rahul.net/aahz/ Androgynous poly kinky vanilla queer het Pythonista I don't really mind a person having the last whine, but I do mind someone else having the last self-righteous whine.

At 05:04 PM 7/31/2001 -0700, Aahz Maruch wrote:
I'm not sure that's the appropriate translation for what perl considers variables and references, but that's OK.
See, Python objects are *global*. Period. No ifs, ands, or buts.
Is this global in the "all variables are in one big pool and can be looked up by name", more or less the equivalent of externing all your C variables, or global in the "Any thread might be able to see it" sense? (I thought Python had lexically scoped variables, but I might be wrong here. I'm an internals guy--the actual language being run is just an implementation detail... :) If it's "All objects are visible to all threads" thing, that's fine, you can still handle threading with some care. Perl's done it wrong once, and I know how to do it much less wrong with Parrot. (It's not, strictly speaking, possible to do it right without the programmer taking an active hand)
Ah. Doable, and not much of a problem. It'll require tweaking the thread model a touch (for which I appreciate you bringing this up) but better now than later.
And, as Paul pointed out, the global interpreter lock only kills you if you're running SMP in the first place.
Yep, well, I do. :) Dan --------------------------------------"it's like this"------------------- Dan Sugalski even samurai dan@sidhe.org have teddy bears and even teddy bears get drunk

Dan Sugalski wrote:
Python certainly has HIGHLY SCOPED variables. Hardly anything is global in that sense. Even a two-line script will not really have truly global, global variables. I wasn't so happy with Aahz's choice of wording there. :) Only the namespace of module names is truly global. i.e. "import sys" gets you the same module object everywhere so you can use modules as containers for other kinds of globals.
Agreed.
Still, I would encourage you to optimize for the common case. People have been predicting multiprocessor would be the common case for years but Moore's law keeps delaying the need to pump up all of our servers with a dozen processors. The Python threading model is one interpreter for all threads unless you really need multiple interpreters in which case you are sort of on your own. -- Take a recipe. Leave a recipe. Python Cookbook! http://www.ActiveState.com/pythoncookbook

Paul Prescod wrote:
No, but it will have global, global objects. Any time one creates an object in Python, it has the potential to be bound to any name. Consider the module foo with attribute bar; foo.bar can be set at any time in any scope to any other object existing somewhere in Python's heap. And that change will be visible everwhere that the name foo.bar is visible. It gets even trickier with multiple threads created in a single module namespace combined with mutable objects and global names (as I illustrated in my response to Dan). -- --- Aahz (@pobox.com) Hugs and backrubs -- I break Rule 6 <*> http://www.rahul.net/aahz/ Androgynous poly kinky vanilla queer het Pythonista I don't really mind a person having the last whine, but I do mind someone else having the last self-righteous whine.

Aahz Maruch wrote:
Right, but what language is otherwise? In what language are variables in one module firewalled from others so that references can't go back and forth? Only functional languages and perhaps security-constrained languages like JavaScript.
I don't see that as having much to do with "globality." Python objects are shared between threads including the module object and the dictionary of module objects. But variables in Python are seldom global in the sense of C's extern or Perl's default scope. -- Take a recipe. Leave a recipe. Python Cookbook! http://www.ActiveState.com/pythoncookbook

Dan Sugalski wrote:
Paul evidently disagrees with me, but I'll write a rebuttal to that right after I finish this. ;-)
Somewhere in between. Python names are scoped; Python objects are not. Here's a trival example: given a global list "l" (and "global" in this phrase almost always means "module global"), and the following code fragment running in one thread def foo(): x = 1 l.append(x) any other threads created in that module will immediately have access to the locally-created object "1". Beyond that, though, given Python's introspection capabilities, it is theoretically correct to say that all variables can be looked up by name.
Yup. Which is why the GIL represents a pretty good compromise, most of the time, particularly the way it's implemented in Python.
Should have added, "...and only if you're running pure Python code." -- --- Aahz (@pobox.com) Hugs and backrubs -- I break Rule 6 <*> http://www.rahul.net/aahz/ Androgynous poly kinky vanilla queer het Pythonista I don't really mind a person having the last whine, but I do mind someone else having the last self-righteous whine.

On Tue, 31 Jul 2001, Eric S. Raymond wrote:
I don't really understand the motivation. Do we want any of those things? Are our current regexps not good enough? What are flexitypes, and why would we really want them? Don't we already have references? Taint might be worth something, and i would see the regexp engine as incrementally better perhaps, but that doesn't seem enough to justify a massive re-engineering of both languages. -- ?!ng

Ka-Ping Yee <ping@lfw.org>:
No, but we want to be able to interoperate with Perl and have if possible have just one back end on which efforts to do things like native code compilation can be concentrated. -- <a href="http://www.tuxedo.org/~esr/">Eric S. Raymond</a> The common argument that crime is caused by poverty is a kind of slander on the poor. -- H. L. Mencken

At 04:52 AM 8/1/2001 +0200, Samuele Pedroni wrote:
This is one of the goals, yes. It's next on the list after getting a working interpreter. There will probably be a TIL version on the road to native compilation.
IMHO also the fact of not merging the type "ontologies" but carrying somehow both around is a bit scary.
It's mildly scary, sure. On the other hand, it's not really any more scary than, say, writing code in COBOL, C++, Basic, or Fortran and calling them from C. Or vice versa. While that's a tad odd, you can pretty easily get used to it, and it works out well. You also don't have to mix languages. That's certainly one of the advantages, but it's far from required. (This is assuming, of course, that there's a performance gain to be had in targeting the parrot back end--if there isn't, then it's reasonably pointless to do so) Dan --------------------------------------"it's like this"------------------- Dan Sugalski even samurai dan@sidhe.org have teddy bears and even teddy bears get drunk

[Dan Sugalski]
If one have to call conversion function from Python/Perl code to deal with the other ontology or deal with unnatural behavior avoiding mixing languages will remain the best thing, also for the problem of reading code in the other language. But for C extensions the situation is a bit different, they should target one ontology (or not?) and then using the extension from both languages will be not that natural. And C extension in Python typically extend the ontology, introduce new types ... regards, Samuele Pedroni.

At 05:51 AM 8/1/2001 +0200, Samuele Pedroni wrote:
I'm not entirely sure there'll be as much of a mismatch between the languages as you might think. One thing that'll help on the perl end is the introduction of optionally typed data--some scalars may be tagged as integer, or arrays as string only, so a lot of the pain of needing to handle conversion and impedance matching will already have been dealt with.
It's possible there'll be a bigger mismatch, but I'm not that sure here, either. (Though I'm stumbling around a bit here, as I've no experience writing Python extensions in C) Some classes of extension will be essentially language neutral--an array's an array's an array and, while some of the details are interesting (like whether it's typed or sized) those are, ultimately details. Important ones, to be sure, but far from an obstacle. New types are a bit more of a challenge. Some things, like declaring methods or overloaded behaviour, are also reasonably small details. Other things, like things I'm not thinking of, are less so. Don't forget I'm also looking at this from the bottom up, not the top down. Parrot's going to be a dynamic language friendly, mildly OO CPU. From that level, these (And I'd get fancy here, but I don't know how to do so without embarrassment in Python): $foo->bar($a, $b, $c) foo.bar(a, b, c) are pretty much syntactically equivalent, and the fact that perl declares methods in packages and handles inheritance on a per-class basis via the @ISA array and Python does something else entirely syntactically is pretty straightforwardly abstractable, and isn't hugely important *for the execution engine* as long as that abstraction is sufficiently powerful. While Perl and Python aren't quite just ALGOL with funny hats, they're really not that different from each other from the level I'm working at. Certainly far less different than they are together from, say, LISP or Forth, or even .NET. (I expect we face similar performance issues with .NET, and the JVM, as neither seem particularly fond of dynamic languages. Though I do see that Jython gets pretty good performance from some of the JIT JVMs) Dan --------------------------------------"it's like this"------------------- Dan Sugalski even samurai dan@sidhe.org have teddy bears and even teddy bears get drunk

Python won't be reengineered. If the Perl VM happens to be able to retargetable for Python, maybe some synergy might happen (like reusable C extensions). But I don't see Python making any compromises in its semantic model, and the new VM would only be used for something like Python 3.0 (or more likely 4.0) if it really offers significant advantages. Dan's recent contributions make me a lot less optimistic. Don't ask me to explain -- I don't have the time, sorry. The other participants in the thread are doing a good job. --Guido van Rossum (home page: http://www.python.org/~guido/)

At 11:00 PM 7/31/2001 -0400, Guido van Rossum wrote:
I would appreciate it if, when you have a chance, you (or someone) would clue me in. I've apparently offended, and that was not, in any way, my intent. My apologies all around. I think I've overstayed my welcome, so I'll bow out here before muddying things up any worse. Dan --------------------------------------"it's like this"------------------- Dan Sugalski even samurai dan@sidhe.org have teddy bears and even teddy bears get drunk

[Guido van Rossum]
[Dan Sugalski]
Ah. My primary job in the Python Community is channeling Guido: By "the other participants are doing a good job", he doesn't mean you *aren't* doing a good job. Not at all! He means that they're doing a good enough job of expressing his concerns that he doesn't feel terrible about not having enough time of his own to participate. No offense taken! You're doing a good job too.
I think I've overstayed my welcome,
Not at all -- it's a good discussion, and perfectly appropriate for Python-Dev. I wish I had more time for it too.
so I'll bow out here before muddying things up any worse.
Please stay. Just try to avoid mentioning Perl, lest you offend Guido again <wink>.

Guido van Rossum writes:
A perl6 value have a vtable, essentially an array of function pointers which comprises the standard operations on that value. I talked to Dan (the perl6 internals guy, dan@sidhe.org) about an impedence mismatch between Perl and Python data types, and he pointed out that you can have Perl values and Python values, each with their own semantics, simply by having separate vtables (and thus separate functions to implement the behaviour of those types). Code can work with either type because the type carries around (in its vtable) the knowledge of how it should behave. Feel free to grill Dan about these things if you want. Nat

On Tue, Jul 31, 2001 at 01:31:01PM -0600, Nathan Torkington wrote:
Python objects all have vtables too (though they're structs, not arrays... I'm not sure why you'd use arrays; check the way Python uses them, you can do just about anything you want with them, including growing them without breaking binary compatibility, due to the fact Python never memmoves/copies) but that wouldn't solve the problem. The problem isn't that the VM wouldn't know what to do with the various types -- it's absolutely problem to make a Python object that behaves like a Perl scalar, or a Perl hash, including the auto-converting bits... The problem is that we'd end up with two different sets of types... Dicts/hashes could be merged, though Perl6 will have to decide if it still wants to auto-stringify the keys (Python dicts can hold any hashable object as key) and arrays could possibly be too, but scalars are a different type. You basically lose the interchangability benifit if Perl6 code all works with the 'Scalar' type, but Python code just uses the distinct int/string/class-instance... But now that I think about it, this might not be a big problem after all. I assume Perl6 will always convert to fit the operation, like Perl5 does. It'll just have to learn to handle a few more objects, and most notably user-defined types and extension types. Python C code already does things like 'PyObject_ToInt' to convert a Python value to a C value it can work with, or just uses the PyObject_<operation> (or PyMapping_<operation>, etc) API to manipulate objects. Python code wouldn't notice the difference unless it did type checks, and the Perl6 types could be made siblings of the Python types to make it pass those, too. We already have the 8-bit and 16-bit strings. About the only *real* problem I see with that is getting the whole farm of mexican jumping beans to figure-skate in unison... It'll be an interesting experience, with a lot of slippery falls and just-in-time recovering... not to mention quite a bit of ego-massaging :-) But I think it's just a manner of typing code and taking the time, and forget about optimizing the code the first couple of years. -- Thomas Wouters <thomas@xs4all.net> Hi! I'm a .signature virus! copy me into your .signature file to help me spread!

Thomas Wouters <thomas@xs4all.net>:
This is just about exactly how I see it, too. The big problem isn't any of the technical challenges -- the discussion so far indicates these are surmountable, and in fact may be less daunting than many of us originally assumed. The big problem will be summoning the political will to make the right commitments and the right compromises. Making this work is going to take strong leadership from Larry and Guido. We're laying some of the technical groundwork now. More will have to be done. But I think the key moment, if it happens, will be the one at which Guido and Larry, each flanked by their three or four chief lieutenants, shake hands for the cameras and issue a joint ukase to their tribes. Tim, hosting that meeting will be your job, of course :-). -- <a href="http://www.tuxedo.org/~esr/">Eric S. Raymond</a> "Those who make peaceful revolution impossible will make violent revolution inevitable." -- John F. Kennedy

Larry, are you following this? What's *your* view? --Guido van Rossum (home page: http://www.python.org/~guido/)

The vtable looks a lot like Python's type object. Is Perl's vtable an object in its own right? In Python, it is: in C, obj->ob_type is the type object, in Python type(obj) or (in 2.2) obj.__class__ gives the type object.
Feel free to grill Dan about these things if you want.
No thanks. :-) --Guido van Rossum (home page: http://www.python.org/~guido/)

At 07:12 PM 7/31/2001 -0400, Guido van Rossum wrote:
Depends on your definition of object, I suppose. There is a mapping from the low-level bits the interpreter needs to interpreted-language level things--while the vtable won't actually be directly manipulatable at the HLL level, manipulating HLL constructs will alter the vtable, so it's effectively the same thing. Dan --------------------------------------"it's like this"------------------- Dan Sugalski even samurai dan@sidhe.org have teddy bears and even teddy bears get drunk

In Python, the type pointer really does point to a full-fledged object. Which probably means something quite different than it means in Perl. :-) --Guido van Rossum (home page: http://www.python.org/~guido/)

At 12:03 AM 8/1/2001 -0400, Guido van Rossum wrote:
Ah. There's going to be a level of indirection there with Parrot. That'll introduce some speed hits, but only when altering the behaviour of the objects. It's been my experience that altering behaviour happens rather less frequently than the behaviour itself happens, so it makes sense to optimize both ends (the interface presented to the HLL programmer and the interface presented to the interpreter engine) and let the middle layer be a bit slower. In this case, the vtable interface doesn't cover general method calls (except to the extent that a variable will have a vtable entry to handle method calls, in case you want to override the inheritance behaviour on a per-object basis, or enable multimethod dispatch for some objects), which are another beast entirely. The vtable interface, at the moment, covers the operators you'd expect to be able to override--assignment, simple math, most string operations, conversion duties, and suchlike things. So this code: a = b + c would call the add vtable method for b, and the assign vtable method for a, while this: a.foo() would make a generic method call on a looking for the foo method, and the vtables generally wouldn't be involved other than handling the (possibly cached) lookup of foo. The point is to put the commonly called things in the vtable in a way that you can avoid as much conditional code as possible, while less common things get dealt with in ways you'd generally expect. (Dynamic lookups with caching and suchlike things) Dan --------------------------------------"it's like this"------------------- Dan Sugalski even samurai dan@sidhe.org have teddy bears and even teddy bears get drunk

But in Python, this is not an operation on a at all! It's an operation on the namespace containing a, and a cannot override it. I'm sure your VM design can accommodate this, but it points out the fundamental difference between the languages in their ideas of what a "variable" is. Python has at least two types of namespaces in this context: some namespaces (like module globals) are dictionaries, others (like function locals) have mapped the variable names to an array of object references. --Guido van Rossum (home page: http://www.python.org/~guido/)

"GvR" == Guido van Rossum <guido@zope.com> writes:
GvR> But in Python, this is not an operation on a at all! It's an GvR> operation on the namespace containing a, and a cannot override GvR> it. I'm sure your VM design can accommodate this, but it GvR> points out the fundamental difference between the languages in GvR> their ideas of what a "variable" is. Python has at least two GvR> types of namespaces in this context: some namespaces (like GvR> module globals) are dictionaries, others (like function locals) GvR> have mapped the variable names to an array of object GvR> references. In the future, I expect module namespaces will be implemented more like function namespaces. That's the "low-hanging fruit" optimization idea we've kicked around for a while. I hope to work on it for 2.3. This discussion of variables also raises a question for the Perl guys: Is there a Perl language reference ala the Python reference manual? Jeremy

At 11:16 AM 8/1/2001 -0400, Jeremy Hylton wrote:
This discussion of variables also raises a question for the Perl guys: Is there a Perl language reference ala the Python reference manual?
Not for perl 6, nope. (Larry's working on that) The Camel and the implementation are pretty much what we've got for perl 5. Dan --------------------------------------"it's like this"------------------- Dan Sugalski even samurai dan@sidhe.org have teddy bears and even teddy bears get drunk

Dan Sugalski wrote:
For Perl5, don't forget perlguts and perlguts illustrated, which may be useful to Pythonistas: http://aspn.activestate.com/ASPN/Products/ActivePerl/lib/Pod/perlguts.html http://gisle.aas.no/perl/illguts/ --david (don't blame me!) ascher

On Wed, Aug 01, 2001 at 10:32:22AM -0700, David Ascher wrote:
Let's kick this in again because people might have missed it: http://www.netthink.co.uk/downloads/ I spent a lot of time working on that, so use it. :)

Dan Sugalski <dan@sidhe.org>:
It sounds like a vtable is very similar to a Python TypeObject, the main difference being that TypeObjects are first-class objects. TypeObjects are immutable, though - you can't use Python code to change one of it's method slots. Whereas, if I understand you correctly, you can use Perl code to change a method in a vtable? Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+

On Tue, 31 Jul 2001, Guido van Rossum <guido@zope.com> wrote:
PUSH "1" PUSH "2" BINARY_ADD In Python that gives "12". In Perl that gives 3. Unless you suggest a PERL_BINARY_ADD and a PYTHON_BINARY_ADD, I don't see how you can around these things. -- gpg --keyserver keyserver.pgp.com --recv-keys 46D01BD6 54C4E1FE Secure (inaccessible): 4BD1 7705 EEC0 260A 7F21 4817 C7FC A636 46D0 1BD6 Insecure (accessible): C5A5 A8FA CA39 AB03 10B8 F116 1713 1BCF 54C4 E1FE Learn Python! http://www.ibiblio.org/obp/thinkCSpy

On Tue, Jul 31, 2001 at 11:10:50PM +0300, Moshe Zadka wrote:
On Tue, 31 Jul 2001, Guido van Rossum <guido@zope.com> wrote:
The Perl version of the compiled code could of course be PUSH "1" COERCE_INT PUSH "2" COERCE_INT BINARY_ADD for Perl's "1" + "2" and PUSH "1" PUSH "2" BINARY_ADD for it's "1" . "2" (or, in the case of variables instead of literals, an explicit 'COERCE_STRING' or whatever.) -- Thomas Wouters <thomas@xs4all.net> Hi! I'm a .signature virus! copy me into your .signature file to help me spread!

On Thu, Aug 02, 2001 at 10:24:14AM -0700, Simon Cozens wrote:
*cough*. Hell no! Addition and string concatenation would be separate opcodes. (I think.)
Note that I later in the same thread change my mind and say that string-concatenation should be a separate opcode :) -- Thomas Wouters <thomas@xs4all.net> Hi! I'm a .signature virus! copy me into your .signature file to help me spread!

Moshe Zadka wrote:
I'm not endorsing the approach but I think the answer is: PUSH PyString("1") PUSH PyString("2") BINARY_ADD versus PUSH PlString("1") PUSH PlString("2") BINARY_ADD i.e. the operators are generic but the operand types vary across languages. So you can completely unify the bytecodes or the types, but trying to unify both seems impossible without changing the semantics of one language or the other quite a bit. -- Take a recipe. Leave a recipe. Python Cookbook! http://www.ActiveState.com/pythoncookbook

[Moshe Zadka]
[Tim]
And if you call Python's + opcode PYTHON_BINARY_ADD and Perl's PERL_BINARY_ADD, it's exactly what I said, isn't it? Are you agreeing or disagreeing with me? ;-) And more importantly, what do you think it means for the feasibility of such a project? -- gpg --keyserver keyserver.pgp.com --recv-keys 46D01BD6 54C4E1FE Secure (inaccessible): 4BD1 7705 EEC0 260A 7F21 4817 C7FC A636 46D0 1BD6 Insecure (accessible): C5A5 A8FA CA39 AB03 10B8 F116 1713 1BCF 54C4 E1FE Learn Python! http://www.ibiblio.org/obp/thinkCSpy

On Wed, Aug 01, 2001 at 11:37:47AM +0300, Moshe Zadka wrote:
He's disagreeing. It's not a PERL vs. PYTHON ADD at all; it's a "string concatenation add" vs. "numerical add". Perl code using the string concatenation operator (apparently, it's not going to be ".", which scares me shitless: Perl6 gets a new string concat operator, but it isn't going to be "+" ? If it is going to be '+', how does it flexitype ?) would use the string-concat add opcode, and Perl code and Python code doing '+' would get the normal BINARY_ADD. The string-to-int conversion in Perl's '+' would be put into the 'Scalar' type. Channeling-Tim--probably-wrong-<wink>-ly y'rs, -- Thomas Wouters <thomas@xs4all.net> Hi! I'm a .signature virus! copy me into your .signature file to help me spread!

On Wed, 1 Aug 2001 10:58:13 +0200, Thomas Wouters <thomas@xs4all.net> wrote:
I think I've lost you. But if you're thinking about Perl compiled to something like PUSH "1" CONVERT_TO_INT PUSH "2" CONVERT_TO_INT BINARY_ADD Then it's only a matter of time before the Perl guys find out it's hella slow and optimize it to: PUSH "1" PUSH "2" CONVERT_TO_INT_TOP_2 BINARY_ADD And then to PUSH "1" PUSH "2" CONVERT_TO_INT_TOP_2_AND_BINARY_ADD And then a simple #define CONVERT_TO_INT_TOP_2_AND_BINARY_ADD PERL_ADD Would finish the 2*PI circle to land exactly where we started with. ;-) -- gpg --keyserver keyserver.pgp.com --recv-keys 46D01BD6 54C4E1FE Secure (inaccessible): 4BD1 7705 EEC0 260A 7F21 4817 C7FC A636 46D0 1BD6 Insecure (accessible): C5A5 A8FA CA39 AB03 10B8 F116 1713 1BCF 54C4 E1FE Learn Python! http://www.ibiblio.org/obp/thinkCSpy

On Wed, Aug 01, 2001 at 12:21:52PM +0300, Moshe Zadka wrote:
I think I've lost you.
But if you're thinking about Perl compiled to something like
No, I'm not. I was, but I'm not. Okay, Moshe, lets try this again. Imagine a Python object, "Scalar", that behaves like a Perl $calar. We're not going to talk about merging the Perl and Python VM here, just about a Python object that behaves somewhat like a Perl scalar, and (for the sake of the discussion) just the string<->int part of the flexitype. The Scalar holds its value as a string, unless it's been stored as an int or 'modified' into an int. Pretend we have an operator "@" to explicitly string-concatenate, exactly like Perl's ".". Now look at this code snippet: one = Scalar("1") two = Scalar("2") three_num = one + two three_s = one @ two "one + two" would simply all BINARY_ADD, and the Scalar type's tp_as_number->nb_add would convert its value internally to an int before handling the operation. "one @ two" would call BINARY_CONCAT (or whatever), which would call a different method to calculate the result. Effectively a different operation. But this isn't "perl add versus python add". "perl add" is just the same as "python add" with a slightly different type, we just get a new opcode for the "perl string concat" operator. Like I (and Eric) said before, there's a lot to be done, and there's bound to be a lot of issues (like how to get *both* scalars to convert their value to a number before arithmatic, above, without resulting to something like PyObject_AsInt() for all binary arithmatic) but those are most likely not insurmountable. -- Thomas Wouters <thomas@xs4all.net> Hi! I'm a .signature virus! copy me into your .signature file to help me spread!

[Guido]
I agree overall, but you picked an unfortunate example. BINARY_DIVIDE is a perfect example: case BINARY_DIVIDE: w = POP(); v = POP(); x = PyNumber_Divide(v, w); Py_DECREF(v); Py_DECREF(w); PUSH(x); if (x != NULL) continue; break; (For Perl'ers, an operation in Python returns a PyObject*, and returns NULL iff the operation wants to raise an exception; the "continue" if x != NULL tells it to go straight back to the top of the eval loop (to fetch the next opcode) if there is no exception; else control flows out of the switch stmt, and masses of code figure out what to *do* with the exception.) That code works unchanged for any pair of objects whatsoever that think they know what to do with an infix "/", since all the intelligence is in PyNumber_Divide(). But BINARY_ADD knows everything there is to know about Python ints, and that's an important speed optimization: case BINARY_ADD: w = POP(); v = POP(); if (PyInt_Check(v) && PyInt_Check(w)) { /* INLINE: int + int */ register long a, b, i; a = PyInt_AS_LONG(v); b = PyInt_AS_LONG(w); i = a + b; if ((i^a) < 0 && (i^b) < 0) { PyErr_SetString(PyExc_OverflowError, "integer addition"); x = NULL; } else x = PyInt_FromLong(i); } else x = PyNumber_Add(v, w); Py_DECREF(v); Py_DECREF(w); PUSH(x); if (x != NULL) continue; break; While we don't peek under the covers often in the eval loop, the places we do were huge benefit/effort wins.

[Tim]
But BINARY_ADD knows everything there is to know about Python ints, and that's an important speed optimization:
Which makes it a perfect example IMO: the abstract *semantics* of the opcode are as neutral as can be, and the *implementation* can grease the path for a common case. A complete redesign can still do that. --Guido van Rossum (home page: http://www.python.org/~guido/)

On Wed, Aug 01, 2001 at 12:42:42AM -0400, Tim Peters wrote:
case BINARY_DIVIDE:
case? Wowsers. Hey, Ruby does that too. We use function pointers, FWIW. Oh, and here's our divide, for comparison: PP(pp_divide) { dSP; dATARGET; tryAMAGICbin(div,opASSIGN); { dPOPPOPnnrl; NV value; if (right == 0.0) DIE(aTHX_ "Illegal division by zero"); value = left / right; PUSHn( value ); RETURN; } } See, this is why we need a new interpreter. :) In slightly more pseudo code: PP(pp_divide) { get_the_stack_pointer; find_somewhere_to_put_result; check_if_overloaded; { NV right = SvNV(POP); NV left = SvNV(POP); NV value; if (right == 0.0) DIE(aTHX_ "Illegal division by zero"); value = left / right; PUSHn( value ); RETURN; } }

Simon Cozens wrote:
People (including me) have tried using threaded code (http://www.complang.tuwien.ac.at/forth/threaded-code.html) and have not measured any significant advantage. I suspect the opcodes to much longer to evaluate than doing the dispatch.
Does every Perl opcode do a function call? Neil

On Thu, Aug 02, 2001 at 10:30:06AM -0700, Simon Cozens wrote:
On Wed, Aug 01, 2001 at 12:42:42AM -0400, Tim Peters wrote:
case BINARY_DIVIDE:
case? Wowsers. Hey, Ruby does that too. We use function pointers, FWIW.
Well, we use *both* :) But function-pointers alone aren't enough, are they ? Or are the bytecodes themselves the function pointers ? *shudder*. I guess you do opcodes as indices into an array-of-function-pointers ? There was talk about doing that for Python too, but I believe it ended up being not significantly faster. Some people have seen bigger speedups just reorganizing the case's in the Big Switch, but those same changes slowed things down on other platforms.
Oh, and here's our divide, for comparison:
In slightly more pseudo code:
Right... Almost the same, except that the check_if_overloaded is a bigger check for Python, and more often true (I suspect.) And, as was said before, assignment isn't a value-filling operation, so instead of 'value = left / right' we have 'PyObject *value = new_object(left / right)' (pseudocode, again) and just push value onto the stack (to be stored into a variable by a subsequent opcode, or just kept in the stack for the running expression.) -- Thomas Wouters <thomas@xs4all.net> Hi! I'm a .signature virus! copy me into your .signature file to help me spread!

Hi, from some recent messages on the jython lists it seems that different people/groups are starting to work on recreating some of the necessary software components in order to port Zope or a create a similar compatible framework on top of Java/Jython. Here's a link to latest related thread: http://aspn.activestate.com/ASPN/Mail/Message/Jython-dev/712337 Jeremy has already chimed in, my personal position is to help them as all other jython users, fixing incompatibilities with CPython and answering to questions. Nothing special. I can imagine that one of the big problem could be restricted execution (if Zope uses that), admitted that any C code functionality can be rewritten in Java. If the PythonLabs have an "official" position/opinion on this and in particular on the last quite speculative message (from Joseph Grace) in the thread, they may post something. I can say something about technical aspects, but I don't feel that I should channel/or invent PythonLabs opinions. regards, Samuele Pedroni.

"SP" == Samuele Pedroni <pedroni@inf.ethz.ch> writes:
SP> my personal position is to help them as all other jython users, SP> fixing incompatibilities with CPython and answering to SP> questions. Nothing special. I can imagine that one of the big SP> problem could be restricted execution (if Zope uses that), SP> admitted that any C code functionality can be rewritten in Java. Zope doesn't use restricted execution. In Zope 2.4, the RestrictedPython project uses the compiler package to generate safe bytecode from Python source code. I presume something similar could be achieved using jythonc, but don't know how hard that would be. SP> If the PythonLabs have an "official" position/opinion on this SP> and in particular on the last quite speculative message (from SP> Joseph Grace) in the thread, they may post something. I can say SP> something about technical aspects, but I don't feel that I SP> should channel/or invent PythonLabs opinions. I don't think there's an official PythonLabs opinion on this in general. If we have one its this: The project(s) -- including the just announced phabric effort -- are worthwhile. We are happy to give advice and make changes to help the projects where we can. I expect the same holds true for the rest of Zope Corp. If there are genuine CPython/Jython incompatibilities that hinder the project, there's no reason not to fix them. The project may also serve as a vehicle to further explore what metaobject protocols should be exported by Jython. Jeremy

[Simon Cozens on Thursday, August 02, 2001 1:30 PM]
Yes, I've tried to read the Perl source before <snarl/wink>. What does "/" mean here? You're using native C arithmetic, or is this a C++ overload? Even simple arithmetic is long-winded in Python, because, for example, the int operations check for overflow, while the float operations allow for catching IEEE-754 exceptions (or SIGFPE in general) and translating them into Python-level exceptions. We also go thru piles and piles of code coercing among different numeric types. If you were to step thru the Python line x = 3.0 / 4 in a debugger, it's a race on Windows between your finger getting numb and the OS crashing ...

On Mon, 30 Jul 2001, "Eric S. Raymond" <esr@thyrsus.com> wrote:
This solution sounds like just taking two VM interpreters and forcing them together by having the first byte of the instruction be "Python opcode" or "Perl opcode". You get none of the wins you were aiming for.
I can see one: garbage collection.
How is GC a problem? Python never promised a specific GC mechanism, so as long as you have something which collects garbage, Python is fine. -- gpg --keyserver keyserver.pgp.com --recv-keys 46D01BD6 54C4E1FE Secure (inaccessible): 4BD1 7705 EEC0 260A 7F21 4817 C7FC A636 46D0 1BD6 Insecure (accessible): C5A5 A8FA CA39 AB03 10B8 F116 1713 1BCF 54C4 E1FE Learn Python! http://www.ibiblio.org/obp/thinkCSpy

On Mon, Jul 30, 2001 at 04:16:49PM -0400, Guido van Rossum wrote:
impact on reusability of the runtime. The bytecode engine cannot be considered independent from the rest of the runtime.
If you must have a portable bytecode format, why not use the JVM? Perhaps it's not optimal, but it works reasonably well, has a few reasonably complete free implementations that are mostly strangling due to lack of manpower, has some support in GCC 3.0, and is actually deployed in browsers and on people's systems *right now*. I fail to see why we should run after some mythical Perl/Python bytecode that would have to be 1) designed 2) implemented 3) debugged 4) actually made available to users 5) actually downloaded by users. (Much the same objections apply to .NET for Unix.) There's also the cultural difference between Python's "write it clearly and then optimize it" and Perl's "let's write clever optimized code right from the start". Perhaps this can be bridged, perhaps not. --amk

"AMK" == Andrew Kuchling <akuchlin@mems-exchange.org> writes:
AMK> On Mon, Jul 30, 2001 at 04:16:49PM -0400, Guido van Rossum AMK> wrote:
impact on reusability of the runtime. The bytecode engine cannot be considered independent from the rest of the runtime.
AMK> If you must have a portable bytecode format, why not use the AMK> JVM? Perhaps it's not optimal, but it works reasonably well, AMK> has a few reasonably complete free implementations that are AMK> mostly strangling due to lack of manpower, has some support in AMK> GCC 3.0, and is actually deployed in browsers and on people's AMK> systems *right now*. I'm not sure I understand the suggestion. The JVM defines an instruction set, but it also defines an entire runtime, right? You've got to live with the JVM's implementation of threads, garbage collection, etc. For the case of Python, that sounds a lot like abandoning CPython and using JPython instead. Or would you suggest using the instruction set but nothing else from the JVM? I'm not sure that there would be much advantage there. If we had a JVM implementation designed to support Python, there would be no need to implement most of the opcodes. We'd only need getstatic and invokevirtual <0.2 wink>. The typed opcodes (int, float, etc.) would never be used. The problem seems to be that the VM ties up a bunch of other issues with the bytecode. Python's VM is intimiately tied up with: - reference counting: each opcode knows when to INCREF and when to DECREF - threads: the global interpreter lock is managed outside the bytecode by the "Do periodic things" code. - object model: BINARY_ADD knows how to special case ints and what method to call to dispatch on all other objects Unless the bytecode is very low level, you buy a lot more than some instructions when you buy an instruction set. Jeremy

Jeremy> If we had a JVM implementation designed to support Python, there Jeremy> would be no need to implement most of the opcodes. We'd only Jeremy> need getstatic and invokevirtual <0.2 wink>. The typed opcodes Jeremy> (int, float, etc.) would never be used. Perhaps Armin Rego's Psyco stuff could make use of them if he chose the JVM as his "other VM". Skip

Hi [Skip Montanaro]
Yes, but feeding the JVM with bytecodes costs more than feeding a real CPU or a VM written to deal quickly with little chunks of code. JVM dynamic loading has a verification phase, accept only full class definitions and then you enter the interpretation/hotspot collecting phase and then dynamic compilation stuff... Samuele.

On Mon, 30 Jul 2001, Andrew Kuchling wrote:
Some of the folks who have done other languages on the JVM have complained about limitations of the Java VM when it comes to supporting features of other languages. Supposedly, Microsoft considered some of those critiques when designing the C# runtime & VM. If, in fact, they have done a better job of generic VM design, then .NET may be worth lookint at. ( Especially as there is now Miguel de Icaza's Mono project. ) ( Of course, politically, that may be inviting a lot of arguments -- see the slashdot threads about whether Mono is a good idea, or is just open source getting suckered by MS! ) -- Steve Majewski

Samuele Pedroni wrote:
I think it is safe to say that the current version of Python.NET is slower than Jython. Now it hasn't been optimized as much as Jython so we might be able to get it as fast as Jython. But I don't think that there is anything in the .NET runtime that makes it a great deal better than the JVM for dynamic languages. The only difference is that Microsoft seems more aware of the problem and may move to correct it whereas I have a feeling that explicit support for our languages would dilute Sun's 100% Java marketing campaign. Also, the .NET CLR is standardized at ECMA so we could (at least in theory!) go to the meetings and try to influence version 2. -- Take a recipe. Leave a recipe. Python Cookbook! http://www.ActiveState.com/pythoncookbook

Also, the .NET CLR is standardized at ECMA so we could (at least in theory!) go to the meetings and try to influence version 2.
Notice the addition "in theory". In practice, this is BS. --Guido van Rossum (home page: http://www.python.org/~guido/)

Guido van Rossum wrote:
It depends on the rules and politics of each particular standards group. It is fundamentally a social activity. It also depends how much effort you are willing to put into promoting your cause. Sam Ruby is chair of the ECMA CLI group. He is a big scripting language fan. http://www2.hursley.ibm.com/tc39/ Also note the presence of Mike Cowlishaw of REXX fame and Dave Raggett of the W3C. Working within a standards body is a gamble. It can pay off big or it can completely fail. We might find Microsoft our strongest ally -- they have always been interested in having the scripting languages work well on their platforms. They would hate to give programmers to have an excuse to stick to Unix or the JVM. I don't personally know enough about this particular circumstance to know whether there is any possibility of significantly influencing version 2 or not. Maybe the gamble isn't worth the effort. But I wouldn't dismiss it out of hand. -- Take a recipe. Leave a recipe. Python Cookbook! http://www.ActiveState.com/pythoncookbook

So it boils down to us vs. MS. Guess who wins whenever there's a disagreement. I still maintain that it's a waste of our time.
Well, your boss has a pact with MS, so AS might pull it off. :-) --Guido van Rossum (home page: http://www.python.org/~guido/)

[Paul Prescod]
Also, the .NET CLR is standardized at ECMA so we could (at least in theory!) go to the meetings and try to influence version 2.
[Guido]
Notice the addition "in theory". In practice, this is BS.
[Paul Prescod]
And cash: being an active member (== the only way to be an *effective* member) of a stds committee is expensive and time-consuming. In my 15-year compiler-geek career, I worked for companies who put major money behind language development, and wrote off several person-years per annum to doing language committee work. Somehow I don't picture your boss springing for that -- which isn't a knock, since I can't picture the people up my chain of command even letting me finish explaining the idea <wink>.
I don't know how ECMA works; indeed, I never heard of it before it agreed to fast-track ECMAScript (nee JavaScript). ANSI/ISO committees work by consensus, and one intransigent member can paralyze the entire effort. Lots of power there! The breadth and depth of compromises made to make progress in the end also account for the "a camel is a horse designed by a committee" outcomes. Note: I was an active member of ANSI X3J17 for a couple of years. This was tasked to come up with a high-level x-language model for expressing parallel computation, plus bindings for a few languages. In order to protect an ancient implementation of Fortran, the IBM representative vetoed any idea that required so much as a runtime *stack* -- or any other dynamic feature. In the end, X3J17 did the honorable thing: it voted to put itself out of its misery <wink>. had-enough-of-that-myself-ly y'rs - tim

Andrew Kuchling writes:
The people designing and implementing perl6 have already agreed on a "do it clean, then make it faster" approach. We can all see the problems with the current Perl internals, and have no desire to repeat the mistakes of the past. There may or may not be impedence mismatch between the two languages (Perl's flexitypes might be one of the sticking points) but this won't be one of them. Nat

"ESR" == Eric S Raymond <esr@thyrsus.com> writes:
ESR> Following my conversation with Guido, I've put doing an ESR> architectural comparison of the existing Python and Perl ESR> bytecodes at the top of my priority list. I'm flying to Taipei ESR> tomorrow and will have a lot of hours on airplanes with my ESR> laptop to do this. Eric, This is a good project. It's really difficult to evaluate the Parrot proposal otherwise. I know quite a bit about Python's VM and runtime, but next to nothing about Perl's. If you're feeling particularly energetic, you might look at some other VM's -- Ocaml, Java, and Ruby come to mind. It is probably a much harder fit for the first two, because they are statically typed. But I'd be quite interested to see a survey of language VM techniques. Jeremy

[Eric S. Raymond]
Don't get too married to that! My bet is that if anyone had time for it, we'd switch the Python VM today to a register model; Skip Montanaro's Rattlesnake project was aiming at that, but fizzled out due to lack of time. The per-opcode fetch-decode-dispatch overhead is very high in SW too, so a register VM can win simply by cutting the number of opcodes needed to accomplish a given bit of useful work. Indeed, eliding SET_LINENO opcodes is the primary reason Python -O runs faster, yet all it saves is one trip around the eval loop per source-code line (the *body* of SET_LINENO is just a test, branch, and store -- it's trivial compared to the overhead of getting to it). Variants of forth-like threading are alternatives to both.

Tim Peters <tim@digicool.com>:
That's an interesting idea. OK, so possibly I was wrong -- I hadn't considered that stack-push/stack-pop operations might introduce overhead comparable to the order-of-magnitude speed difference between registers and main memory in hardware. I'm still skeptical, but my mind is open. -- <a href="http://www.tuxedo.org/~esr/">Eric S. Raymond</a> You know why there's a Second Amendment? In case the government fails to follow the first one. -- Rush Limbaugh, in a moment of unaccustomed profundity 17 Aug 1993

"Eric" == Eric S Raymond <esr@thyrsus.com> writes:
Eric> Tim Peters <tim@digicool.com>: >> The per-opcode fetch-decode-dispatch overhead is very high in SW too, >> so a register VM can win simply by cutting the number of opcodes >> needed to accomplish a given bit of useful work. Eric> That's an interesting idea. OK, so possibly I was wrong -- I Eric> hadn't considered that stack-push/stack-pop operations might Eric> introduce overhead comparable to the order-of-magnitude speed Eric> difference between registers and main memory in hardware. I'm Eric> still skeptical, but my mind is open. Order of magnitude increases? Maybe, maybe not. Still, something like ADD a1,a2,a3 is going to be faster than PUSH a1 PUSH a2 ADD POP a3 My original aim in considering a register-based VM was that it is easier to track data flow and thus optimize out or rearrange operations to reduce the operation count. Translating Python's stack-oriented VM into a register-oriented one was fairly straightforward (at least it was back when I was fiddling with it - pre-1.5). The main stumbling block was that pesky "from module import *" statement. It could push an unknown quantity of stuff onto the stack, thus killing my attempts to track the location of objects on the stack at compile time. Skip

Skip Montanaro <skip@pobox.com>:
Are you *sure* about that? I'm pretty certain it can't be true, since the compiler has to know at all times how much is on the stack, so it can decide how much stack space is needed. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+

On Tue, Jul 31, 2001 at 10:51:35AM +1200, Greg Ewing wrote:
Skip Montanaro <skip@pobox.com>:
I think Skip meant it does an arbitrary number of load-onto-stack store-into-namespace operations. Skip, you'll be glad to know that's no longer true :) Since 2.0 (or when was it that we introduced 'import as' ?) import-* is not a special case of 'IMPORT_FROM', but rather a separate opcode that doesn't touch the stack. 'IMPORT_FROM' is now only used to push a given name from TOS onto the stack:
Bloody hell, what's that LOAD_CONST doing there ? I think I found a bug ;P Sigh... Sleep first, fix later. -- Thomas Wouters <thomas@xs4all.net> Hi! I'm a .signature virus! copy me into your .signature file to help me spread!

Skip> The main stumbling block was that pesky "from module import *" Skip> statement. It could push an unknown quantity of stuff onto the Skip> stack Greg> Are you *sure* about that? I'm pretty certain it can't be true, Greg> since the compiler has to know at all times how much is on the Greg> stack, so it can decide how much stack space is needed. Thomas> I think Skip meant it does an arbitrary number of Thomas> load-onto-stack Thomas> store-into-namespace Thomas> operations. Skip, you'll be glad to know that's no longer true Thomas> :) Since 2.0 (or when was it that we introduced 'import as' ?) Thomas> import-* is not a special case of 'IMPORT_FROM', but rather a Thomas> separate opcode that doesn't touch the stack. I'm not sure what I meant any more. (They say eye witness testimony in a courtroom is quite unreliable.) I'm pretty sure Greg's analysis is at least partly correct (in that that couldn't have been why I failed to implement a converter for IMPORT_FROM). I went back and looked briefly at my old code last night (which was broken when I put it aside - don't *ever* do that!) and could find nothing that would indicate why I didn't like "from-import-*". The instruction set converter would refuse to try converting any code that contained these opcdes: {LOAD,STORE,DELETE}_NAME, SETUP_{FINALLY,EXCEPT}, or IMPORT_FROM. At this point in time I'm not sure which of those six opcodes were just ones I hadn't gotten around to writing converters for and which were showstoppers. wish-i-had-more-time-for-this-ly y'rs, Skip

Obviously, just as the new design is aiming at Perl 6, it would be aiming at Python 3. Nothing's impossible these days, so I am keeping an open mind. I expect that in addition to the bytecode, the entire runtime architecture would have to be shared though for this to make sense, and I'm not sure how easy that would be, even if Perl is willing to be flexible. Most of Python's run-time semantics are very carefully defined and shouldn't be changed in order to fit in the common runtime. I'm looking forward to Eric's comparison of the two run-time systems. (Eric, be sure to use a copy of 2.2a1 or the descr-branch -- *don't* use the CVS trunk.) --Guido van Rossum (home page: http://www.python.org/~guido/)

Guido van Rossum <guido@zope.com>:
What would the CVS magic invocation for that be? And...um...why? Has the bytecode changed significantly recently? -- <a href="http://www.tuxedo.org/~esr/">Eric S. Raymond</a> The spirit of resistance to government is so valuable on certain occasions, that I wish it always to be kept alive. It will often be exercised when wrong, but better so than not to be exercised at all. I like a little rebellion now and then. -- Thomas Jefferson, letter to Abigail Adams, 1787

What would the CVS magic invocation for that be?
cvs update -r descr-branch or cvs checkout -r descr-branch python/dist/src Or just download 2.2a1.
And...um...why? Has the bytecode changed significantly recently?
Not the bytecode, but the rest of the runtime has changed tremendously, and as I tried to explain over the phone, that has a big impact on reusability of the runtime. The bytecode engine cannot be considered independent from the rest of the runtime. --Guido van Rossum (home page: http://www.python.org/~guido/)

Guido van Rossum <guido@zope.com>:
Or just download 2.2a1.
It's cool. My local installation is from 2.2.a0. I'll update.
OK, let's try to factor this design problem. Let's suppose, for the sake of the design discussion, that we can make the type ontologies of the Perl and Python bytecode match up. (Note: making the type ontologies of the two bytecodes match is not the same problem as making the type ontologies of the *languages* match up. It should be rather simpler because a lot of the differences between, e.g., class semantics can probably be compiled away. Not a trivial problem, but humor me.) Let's further suppose that we have a callout mechanism from the Parrot interpreter core to the Perl or Python runtime's C level that can pass out Python/Perl types and return them. Given these two premises, what other problems are there? I can see one: garbage collection. What others are there? -- <a href="http://www.tuxedo.org/~esr/">Eric S. Raymond</a> An armed society is a polite society. Manners are good when one may have to back up his acts with his life. -- Robert A. Heinlein, "Beyond This Horizon", 1942

"ESR" == Eric S Raymond <esr@thyrsus.com> writes:
ESR> Let's suppose, for the sake of the design discussion, that we ESR> can make the type ontologies of the Perl and Python bytecode ESR> match up. What is a type ontology? The definition of ontology I'm familiar with is too broad to be useful in understanding what you're getting at. I've never heard the technical term "type ontology". ESR> (Note: making the type ontologies of the two bytecodes match is ESR> not the same problem as making the type ontologies of the ESR> *languages* match up. It should be rather simpler because a ESR> lot of the differences between, e.g., class semantics can ESR> probably be compiled away. Not a trivial problem, but humor ESR> me.) If I guess at what you mean-- a fuzzy notion that the underlying type system can support both languages-- then I submit that most of the hard problems are indeed here. ESR> Let's further suppose that we have a callout mechanism from the ESR> Parrot interpreter core to the Perl or Python runtime's C level ESR> that can pass out Python/Perl types and return them. Not quite sure what you mean ehre. ESR> Given these two premises, what other problems are there? ESR> I can see one: garbage collection. What others are there? I think you mean memory management in general, not just GC. Others: thread model, interpreter management (such as creating embedded interpreter objects). Jeremy

Jeremy Hylton <jeremy@zope.com>:
I first heard it in connection with cross-language RPC. The "type ontology" of a language or protocol is its implicit theory of what kinds of things there are in the universe. It's actually a pretty reasonable specialization of the term "ontology" in philosophy. -- <a href="http://www.tuxedo.org/~esr/">Eric S. Raymond</a> A man who has nothing which he is willing to fight for, nothing which he cares about more than he does about his personal safety, is a miserable creature who has no chance of being free, unless made and kept so by the exertions of better men than himself. -- John Stuart Mill, writing on the U.S. Civil War in 1862

On Mon, Jul 30, 2001 at 05:18:31AM -0400, Eric S. Raymond wrote:
Let's suppose, for the sake of the design discussion, that we can make the type ontologies of the Perl and Python bytecode match up.
I'm afraid I'll have to side with Jeremy when I say, "What?"
Given these two premises, what other problems are there?
I can see one: garbage collection. What others are there?
As a midnight braindump: What about Perl's 'dynamic' (or 'really JIT') compilation ? The incessant weak typing -- would this be part of the Perl side of Parrot, or part of the Parrot types ? The differences in the regex engine; in Python, regular expressions are optional. Also, the Perl engine has some features SRE hasn't, yet, and vice versa (last I checked, Perl's regexps didn't do unicode or named groups.) And what about Perl's 'Taint' mode ? I don't see how you can emulate that ontop of the Parrot runtime, as it's a tag that gets carried into operations. And I won't even start with Perl's more archaic features, that change the whole working of the interpreter. You mentioned regular expressions as an upside for Python, from this 'merger'. Why is that ? We have a good regex engine, and it's tuned to Python's needs. Do we need 'regex literals' ? Why ? And why would we need a merger with Perl for that, anyway -- I've seen some arbitrary-type-literals suggestions come by in the last couple of days that would make it possible :-) -- Thomas Wouters <thomas@xs4all.net> Hi! I'm a .signature virus! copy me into your .signature file to help me spread!

Thomas Wouters <thomas@xs4all.net>:
Explained in public reply to Jeremy.
You mentioned regular expressions as an upside for Python, from this 'merger'. Why is that ?
No. I was referring to the fact that we have *already* coopted Perl's regexp design. -- <a href="http://www.tuxedo.org/~esr/">Eric S. Raymond</a> The end move in politics is always to pick up a gun. -- R. Buckminster Fuller

On Tue, Jul 31, 2001 at 01:24:32AM +0200, Thomas Wouters wrote:
Parrot types ? The differences in the regex engine; in Python, regular expressions are optional. Also, the Perl engine has some features SRE
If regex opcodes form part of the basic VM, would the main loop end up looking like the union of ceval.c and pypcre.c/_sre.c? The thought is too ghastly to contemplate, though a little part of me [*] would like to see it. --amk [*] "Taking it in its deepest sense, the shadow is the invisible saurian tail that man still drags behind him. Carefully amputated, it becomes the healing serpent of the mysteries. Only monkeys parade with it." C.G. Jung, in _The Integration of the Personality_. (1939)

Andrew Kuchling writes:
(perl guy speaking alert) The plan for perl6 is to implement the regular expression engine as opcodes. We feel this would be cleaner and faster than having the essentially separate module that we have right now. I think our current perl5 project manager was the one who said that we have no idea how inefficient our current RE engine is, because it's been "optimized" to the point where it's impossible to read. The core loop would just be the usual opcode dispatch loop ("call the function for the current operation, which returns the next operation"). The only difference is that some of the opcodes would be specific to RE matches. (I'm unclear on how much special logic RE opcodes involve--it may be possible to implement REs with the operations that regular language features like loops and tests require). Nat

On Mon, Jul 30, 2001 at 08:08:47PM -0700, Nathan Torkington wrote:
The big difference I see between regex opcodes and language opcodes is that regexes need to backtrack and language ones don't. Unless the idea is to compile a regex to actual VM code similar to that generated by Python/Perl code, but then wouldn't that sacrifice efficiency? --amk

At 08:01 AM 7/31/2001 -0400, Andrew Kuchling wrote:
Backtracking in regexes isn't all that much different from some of the things you do at the language level, and certainly it's a straightforward mapping of regexes onto a series of string & integer register manipulations with simple branch-and-test code, with perhaps some stack space thrown in for deeply nested regexes.
The idea is (currently, barring bad performance) to compile down to VM code similar to what Ptyhon or perl would generate. There's the potential to slow things down, which I'm rather nervous about. On the other hand, the sort of code that would get emitted for regexes maps very well onto the instruction sets of hardware CPUs, so you'd get a corresponding boost in speed once translated to machine code. (Though granted you'd need to be able to do the conversion on the fly, as the regex itself isn't guaranteed to be known at compile time) Dan --------------------------------------"it's like this"------------------- Dan Sugalski even samurai dan@sidhe.org have teddy bears and even teddy bears get drunk

Dan, Have you looked at the byte code used in SRE, Python's (mostly) Perl-compatible regex package? Worth a look! --Guido van Rossum (home page: http://www.python.org/~guido/)

At 11:46 PM 7/31/2001 -0400, Guido van Rossum wrote:
Have you looked at the byte code used in SRE, Python's (mostly) Perl-compatible regex package? Worth a look!
Nope, not yet, but I will. (You folks will end up making a Python programmer out of me yet, I expect... :) This and the C extension interface are the two big Python things I've got on my "must peer deeply into" list. Are there other places I should be concentrating on? (I'm poking about with the interpreter loop and associated code, but that's the sort of thing an expert with 20 minutes free will get me much farther than a week or two with the source) Dan --------------------------------------"it's like this"------------------- Dan Sugalski even samurai dan@sidhe.org have teddy bears and even teddy bears get drunk

On Tue, Jul 31, 2001 at 11:52:01PM -0400, Dan Sugalski wrote:
Nope, not yet, but I will. (You folks will end up making a Python programmer out of me yet, I expect... :)
Well, at least you know it's happening... most people don't realize until it's too late :)
What do you say I write up a longish email explaining quickly how Python objects work, and how the Python interpreter and extension code works with them ? I think it'd help with getting us talking about the same thing when we say 'interpreter' or 'type' :-) And the beauty of my being in a different timezone is that noone else is posting right now, so I can take the time without someone else doing it spread out over four posts in separate subthreads :) -- Thomas Wouters <thomas@xs4all.net> Hi! I'm a .signature virus! copy me into your .signature file to help me spread!

On Tue, Jul 31, 2001 at 11:52:01PM -0400, Dan Sugalski wrote:
[SRE] and the C extension interface are the two big Python things I've got on my "must peer deeply into" list.
I can't say much about SRE, but the C extension interface should be on anyone's "must peer deeply into" list, except that there isn't a real need to peer deeply into it :) Lets peer into it. I'm going to discuss the pre-2.2 state of affairs, mostly because I know jack shit about the 2.2-type-dichotomy-healing :) In case you aren't aware, we are currently into the 2.2 alphas, where this story is slightly different because you can now subclass types. If Guido or anyone else involved in the descr-branch wants to tune in and correct the explanation for Python 2.2, that's fine by me. As you might have heard, everything in Python is an object. This is true even for the C side of things. A Python object, any Python object, is essentially a struct like this: stuct PyExample { int ob_refcnt; /* reference count */ struct _typeobject *ob_type; /* type of this object */ /* type-specific data goes here */ } The common header is made available as a #define, so that we can add extra info in severe-debug mode (though this breaks binary compatibility, so it should not be done in production builds.) For instance, the struct to hold a (normal, unbounded) Python integer is typedef struct { PyObject_HEAD long ob_ival; } PyIntObject; The struct _typeobject pointer is a pointer to a Type object, which also acts as a vtable. It contains all the information about the type, including how to operate on it. The typeobject is itself also an object! Its type is the PyType type, which is its own type. The typeobject is actually what you get back when you do 'type(object)' in Python code. All this is also explained in Include/object.h, by the way. The type object holds things like the type name, how to print it, howmuch memory should be allocated, a couple of flags to implement binary compatibility, and a whole boatload of function pointers. The function pointers for operations are spread out over the type struct and several separate structs, to which the type struct points: tp_as_number, tp_as_sequence, and tp_as_mapping. Each of these can be NULL if the type doesn't implement the operations. Some of the operations that are defined on the type struct itself are get-attribute, set-attribute, comparison, get-hashvalue, convert-to-string and call-as-function. The PyNumberMethods struct (tp_as_number) contains arithmatic operations (such as adding, mulitplying, shifting, XORing, and in-place variations of each for use with augmented assignment), truth-value-testing, and converting to various other integer types. The PySequenceMethods struct (tp_as_sequence) contains sequence operations such as membership-test, item- and slice-retrieval and -assignment, and sequence concatenation and repetition. The latter two are somewhat of an odd duck, since they implement the same Python operation ("+" and "*") as the 'add' and 'multiply' methods from the PyNumberMethods struct. They're mainly so the Python runtime can properly handle "number * list" as well as "list * number" without attempting to convert list to an integer. Lastly there's PyMappingMethods. I'm not at all sure why this is a separate sturct, as the three operations it contains (length inquiry, item-retrieval and item-assignment) are also covered by the PySequenceMethods, but I suspect some kind of hysterical raisin crept in here :) To see this construct in action, take a look at Objects/intobject.c, in the Python source tree (doesn't really matter which version.) It defines a bunch of functions to do the arithmatic, fills a PyNumberMethods struct with the appropriate pointers (leaving unsupported ones NULL), and fills a new PyTypeObject with the right info for the "int" type. To contrast, the PyFunction type is nearly empty. It defines getattr and setattr, and a 'repr' function (convert to string). You can't compare it (it will always compare false to anything but itself), add it, index it, or get its length, but you can get attributes from it. And the average function has some interesting attributes: you can, from Python code and C code alike, easily get at things like the bytecode it consists of, the number of arguments it expects, the names of those arguments, etc. Each of those is, of course, a Python object in its own right :) Function objects are defined in Objects/funcobject.c, though the actual execution of function objects is done elsewhere. An 'instance' object is a slightly different case in this story (more so in the post-2.2 story, but I'm not sure howmuch more.) An instance object is a Python class, defined in Python code by the user/programmer. It can choose to implement any of the above operations by defining magically named methods on the class. For instance, to implement the 'getitem' protocol, it defines a function __getitem__, and to implement addition, __add__. The Type object for instance objects acts as a proxy for the Python class, though this does represent some problems: instance objects need to be treated in some areas of the interpreter (but this is almost certain to have been changed in 2.2.) But back to generic Python objects. Obviously, when writing C code, you don't want to manipulate Python objects by directly accessing all those layers of C structs all the time. So each builtin type defines useful C functions and macros to help. For instance, to create a new list, you call PyList_New(size). To test whether something is a list, PyList_Check(object). To index a list, PyList_GetItem(list, index), or a macro version of that, PyList_GET_ITEM. The macro version doesn't do type or boundary checks, so it should only be used when you already did those checks. Likewise, there is PyInt_AsLong (convert a PyInt to a C 'long') and PyInt_AS_LONG. But even that isn't that helpful in a dynamically typed language, since you have to know an object's type in order to call the type-specific functions. So we also have two more abstraction layers. The most abstract one is the Object layer. Basically, regardless of the type of object, you can always manipulate it using PyObject_*() functions. Some PyObject_ functions have a direct relation to Python code, others are only useful to C code. There are PyObject_ functions such as PyObject_IsTrue, PyObject_Compare, PyObject_GetAttr, PyObject_CallObject, etcetera. All these work regardless of the type. If the type (or instance) does not implement the operation, an exception is raised (see below.) The other layer is the slightly less abstract layer that differentiates in the same way as the PyNumberMethods, PySequenceMethods, PyMappingMethods structs. To add two objects, you call PyNumber_Add(o1, o2). To explicitly concatenate two objects, you call PySequence_Concat(o1, o2). All of these work for any type that defines the behaviour, including instance types. Note that calling one of these functions can end up executing arbitrary Python code, so they shouldn't be called in any time-critical situation :-) As a last part of the API, there is the reference counting. As I showed a couple of pages back, all objects contain a reference count. Operations that copy a reference should increment the reference count, using Py_INCREF, and deleting a reference should be done using Py_DECREF. The latter also does the deallocation and cleanup if the DECREF causes the refcount to drop to 0. All API functions that return a reference define whether they return a new reference, which the caller has to DECREF when throwing it away, or a borrowed reference, which shouldn't be DECREF'ed, or kept around without INCREFing it. Exceptions, by the way, are set by setting a global (or thread-local, in a threaded interpreter) variable to contain the Python exception object, and then returning an error flag (either NULL, -1 or 0, depending on the function) to the caller. All code should check all return values and always return an error value if a called function does so, unless it's prepared to handle the exception itself. To compare errors with a particular error 'class', there is PyErr_ExceptionMatches (which also handles 'subclasses' of exception types.) If the error value returned by a particular function is also a real possible value (such as the value returned by PyNumber_AsLong), there is PyErr_Occured() to see if it was a real exception, or a legitimate -1 value. So that's it for the API. All Python objects can be manipulated, created and destroyed this way. Most extension types provide their functionality either in builtin-operations (in the type object) or by providing methods that can be called (using PyObject_GetAttr and PyObject_CallObject), and only rarely need to export an explicit API of their own. (And as you probably know, exporting an API from a shared library to another shared library is... slightly tricky :) Now, the Python bytecode-interpreter itself is very small. All it does is interpret bytecode, and then call out to the API to make it do the actual work. It has some special knowledge about some types, for optimzation, and it does all the real exception handling and namespace handling (name lookup and such), but the actual manipulation of objects is left to the API. Even so, it has enough to do. In the case of exceptions, it has to search up the function for a 'catching' block (a try/except or try/finally statement), or break out of the function to let the error percolate to the caller. Namespaces are a complete story in their own right. Python has three distinct namespaces: the 'local' namespace inside a function (or 'code block'), the 'global' or 'module-level' namespace at the toplevel of a file, and the 'builtin' namespace which holds builtin functions and names. The search order is always local->global->builtin, though for code that is executed at the module level (not inside a function), local is the same as global. In Python, variables are defined by assignment. As a consequence, any variable that is assigned to in a code block, is local to that block. Any variable that isn't assigned to but is just referenced thus has to be defined in the global namespace. The compiler keeps track of this, and generates 'LOAD_FAST' (for loading from the local namespace, inside a function), 'LOAD_GLOBAL' (for loading from the global or builtin namespace, skipping the local namespace), and 'LOAD_NAME' (for occurances where it's truly not known where a variable comes from, and both local and global namespaces have to be searched.) The latter is necessary because Python has a few operations that can modify a local namespace at runtime (such as 'from module import *', and 'exec'.) Since Python 2.1, Python has 'nested scopes', which makes code blocks defined inside other code blocks slightly special: the 'parent' scope is also searched for any variable that isn't local to the nested scope. Most of this, but not all, is done at compiletime, IIRC. (Jeremy can correct me if I'm wrong :) I tried to make the distinction between a 'function' and a 'code block', since the VM really deals with the second, not the first. I suspect that the current Python VM's mechanics to deal with nested scopes could be generalized so that Perl's lexical closures could fit as well (I assume that won't change in perl6 :). Then again, Perl already knows where a variable comes from, so the VM will probably want to put that to good use... The main reason Python does the name-searching at runtime is that a global namespace can be altered from outside the currently-executing codeblock (threads, or other modules, can do 'module.range = my_fake_range', and suddenly the range() function does something completely different throughout the entire module.) A good overview of how the VM uses the API can be obtained by looking at Python/ceval.c, and looking for "switch (opcode)". You'll end up at the top of the main bytecode switch. Most of the opcode names are pretty descriptive, but if you want a better overview, take a look at the 'dis' module's documentation. (www.python.org is still down, but you can use one of the mirrors: http://python.mirrors.netnumina.com/doc/current/lib/module-dis.html http://python.mirrors.netnumina.com/doc/current/lib/bytecodes.html And an easy way to see how a Python operation is actually implemented is using the 'dis' module interactively:
3 SET_LINENO 2 6 SETUP_LOOP 44 (to 53) >> 9 SET_LINENO 2 12 LOAD_FAST 0 (x) 15 JUMP_IF_FALSE 33 (to 51) 18 POP_TOP 19 SET_LINENO 3 22 LOAD_CONST 1 ('Hello ') 25 LOAD_CONST 2 ('Perl ') 28 BINARY_ADD 29 LOAD_CONST 3 ('World!') 32 BINARY_ADD 33 PRINT_ITEM 34 PRINT_NEWLINE 35 SET_LINENO 4 38 LOAD_FAST 0 (x) 41 LOAD_CONST 4 (1) 44 INPLACE_SUBTRACT 45 STORE_FAST 0 (x) 48 JUMP_ABSOLUTE 9 >> 51 POP_TOP 52 POP_BLOCK >> 53 LOAD_CONST 0 (None) 56 RETURN_VALUE Keep in mind that this is stack-based, and all operations manipulate the stack. (for instance, LOAD_FAST pushes the variable onto the stack, and BINARY_ADD pops two values, adds them, and pushes the result.) -- Thomas Wouters <thomas@xs4all.net> Hi! I'm a .signature virus! copy me into your .signature file to help me spread!

On Wed, Aug 01, 2001 at 03:36:07PM +0200, Thomas Wouters wrote:
This looks rather like a Perl SV (scalar value): struct STRUCT_SV { /* struct sv { */ void* sv_any; /* pointer to something */ U32 sv_refcnt; /* how many references to us */ U32 sv_flags; /* what we are */ };
We use double indirection so that we can switch between types very quickly; the sv_any pointer points to another structure or an integer/float.
The struct _typeobject pointer is a pointer to a Type object, which also acts as a vtable.
That's sounding more like how we plan to do things for Perl 6.
% ./perl -Dts -e 'print "Hello Python people!\n"' => (-e:1) null => (-e:1) const(PV("Hello Python people!\12"\0)) => PV("Hello Python people!\12"\0) (-e:1) stringify [ That was constant folding, BTW. - SC] EXECUTING... => (-e:0) enter => (-e:0) nextstate => (-e:1) pushmark => * (-e:1) const(PV("Hello Python people!\12"\0)) => * PV("Hello Python people!\12"\0) (-e:1) print Hello Python people! => SV_YES (-e:1) leave perl -MO=Bytecode,-S -e 'print "Hello Python people!\n"' will give you a complete disassembly of the bytecode, but that's the version that can be serialised to disk, so it's a lot more verbose than it need be - it sets up all the opcodes, their flags, their sequence numbers, their connections to other nodes in the opcode tree and so on. If you really want to get stuck in, read the internals tutorial at http://www.netthink.co.uk/downloads/ Simon

At 03:36 PM 8/1/2001 +0200, Thomas Wouters wrote:
Ah, I see. (and thanks heaps) Well, I just rejigged my thinking a bit on what we need to export, then. My take's been both more agressively black-box about things, and much less OO. Too much C on the brain, I expect. Dan --------------------------------------"it's like this"------------------- Dan Sugalski even samurai dan@sidhe.org have teddy bears and even teddy bears get drunk

Andrew Kuchling wrote:
Welcome to Perl. :) I don't really understand it but here are references that might help: http://aspn.activestate.com/ASPN/Mail/Message/638953 http://aspn.activestate.com/ASPN/Mail/Message/639000 http://aspn.activestate.com/ASPN/Mail/Message/639048 -- Take a recipe. Leave a recipe. Python Cookbook! http://www.ActiveState.com/pythoncookbook

Thomas Wouters writes:
Perl's REs now do Unicode. Perl 6's REs will do named groups.
And I won't even start with Perl's more archaic features, that change the whole working of the interpreter.
Those are going away. Perl people hate them as much as you do--the only time they're used now is to make deliberately hideous code, and hardly anyone will seriously lament the passing of that ability. No more "change the starting position for subscripts", no more "change all RE matches globally", and so on. Nat

On Mon, Jul 30, 2001 at 08:12:04PM -0700, Nathan Torkington wrote:
Yeah, I thought as much, which is why I wasn't going to start on them :)
I don't really hate the features, I just don't use them, and wouldn't want them in Python :-) I do actually program Perl, and will do a lot more of it in the next couple of months at least (I switched projects at work, to one that will entail Perl programming roughly 80% of the time) -- I just like Python a lot more. Your comments do lead me to ask this question, though (and forgive me if it comes over as the arrogant ranting of a Python bigot; it's definately not intended as such, even though I only have a Python-implementors point of view.) What's going to be the difference between Perl6 and Python ? The variable typing-naming ($var, %var, etc) I presume, and the curly bracket vs. indentation blocking issue. Regex-literals, 'unless', the '<expression> if/unless/while <boolean exp>' shortcut, I guess ? Those are basically all parser/compiler issues, so shouldn't be a real problem. The transmorphic typing is trickier, as is taint mode and Perl's scoping rules.... Though the latter could be done if we refactor the namespace-creation that is currently done implicitly on function-creation, and allow it to be done explicitly. The same goes for the variable-filling-assignment (which is quite different from the name-binding assignment Python has.) I don't really doubt that Perl and Python could use the same VM.... I'm not entirely certain howmuch of the shared VM the two implementations would actually be using. Is it worth it if the overlap is a mere, say, 25% ? (I think it's more, but it depends entirely on howmuch different Perl6 is from Perl5, and howmuch Python is willing to change.... Lurkers here know I'm agressively against gratuitous breakage :) -- Thomas Wouters <thomas@xs4all.net> Hi! I'm a .signature virus! copy me into your .signature file to help me spread!

One of the things I picked up from the Perl conference is that Perl users *seem* (to me) to have a higher tolerance for code breakage than Python users. (and Python users have a higher tolerance than (let's say) Java users) Even if we put aside Perl 6, Perlers talk pretty glibly about ripping little used features out in Perl 5.8.0 and Perl 5.10 and so forth. e.g. Damian said that Autoload is going away (or pseudo hashes or something like that). Whether or not he was right, nobody in the room threw tomatoes as I'm sure they would if Guido tried to kill __getattr__. Admittedly, I never know when I hear stuff like "tr///CU is dead" or "package; is dead" whether each was a feature that has been in for three years or was added to an experimental release and removed from the next experimental release. I'm not criticizing the Perl community. Acceptance of change is a good thing! But I think they should know how conservative the Python world is. Last week there were storm troopers heading for Guidos house when he announced that the division operator is going to change its behaviour two or three years. That means it would take a major PR effort to convince the Python community that even minor language changes would be worth the benefit of sharing a VM. -- Take a recipe. Leave a recipe. Python Cookbook! http://www.ActiveState.com/pythoncookbook

Paul Prescod writes:
(off-topic, but he started it!) That's a really interesting comment. In many ways it's probably true, although make no mistake about it--we take it in the neck for every language change we make! The features we're talking about removing are either deprecated or experimental. With that said, we've probably been less rigorous than we should be about segregating experimental features from blessed ones. I suspect that because experimental features are the only way to do something, they still get used no matter how much we caution that they may go away. I've been eyeing your 'import from the future' system of getting experimental features. I'll be interested to see how this holds up. Perl and Python believe in active maintenance and development with many small incremental releases (unlike Java with occasional bursty releases, and C++ and C which are to all intents and purposes static). Changes to the language cause problems we share, and the other mainstream languages don't provide us with any direction on how to approach the problem. We in the Perl camp have bandied around the idea of "use perl 5.004" as a way to get all the behaviour of the 5.004 release. That's a noble idea, but forces us to keep a lot of crap around when one of the goals of active development is to be able to shed obstructive missteps in language evolution. Nat

On Tue, Jul 31, 2001 at 08:26:47PM -0600, Nathan Torkington wrote:
I've been eyeing your 'import from the future' system of getting experimental features. I'll be interested to see how this holds up.
It's really not that different form Perl's "use". They're both scanned for by the parser/tokenizer, they're both used to change the semantics of an operation, and they also double as a normal 'get and use a package' operation (which is, in fact, the primary task of 'import' and 'use' both, right ? :) The main difference is that changing the behaviour of the Python interpreter is a lot less common than changing the behaviour of the Perl one, and Guido wants to keep it that way. The purpously 'magic' spelling of the future-import statement is part of that. -- Thomas Wouters <thomas@xs4all.net> Hi! I'm a .signature virus! copy me into your .signature file to help me spread!

Nathan Torkington wrote:
Actually, Python doesn't usually have experimental features. "import from future" is about introducing new features that will break code "early". On the one hand, some people want the feature as soon as possible. On the other, people are afraid of the code breakage (e.g. adding a keyword in Python breaks code) and need time to adjust. The future construct is the compromise. Python has no real mechanism for experimental features other than documentation, patches and so forth. I can only imagine one or two experimental features in the history of the language, though.
Yes, Guido has rejected that idea for Python as too much effort. Imagine the regression testing problems alone! -- Take a recipe. Leave a recipe. Python Cookbook! http://www.ActiveState.com/pythoncookbook

Since python does so much by name (I think locals are the only place where name lookup is compiled away), dictionary lookup is pretty fundamental to the runtime, and is used by a number of opcodes. No reason that dictionary lookup couldn't be part of the common runtime, or else tucked behind an abstract interface common to Python and Perl lookups, but it's something to keep in mind if the VM is going to operate on a similar level to the current one. But then, I've always thought that one of the problems with trying to optimize Python was that the VM was too high level. If some sort of Forth-like extensible threaded code were used, you could build the current opcodes from lower level primitives. Re: stack vs. register machines: Some Forth implementations cache some of the top of stack in registers, but the more you try to cache, the hairier it gets. ( But you can figure out the bookkeeping once and automatically generate the code variations. ) You might take a look at Anton Ertl's VMGEN: <http://www.complang.tuwien.ac.at/anton/vmgen/> | Vmgen generates much of the code for efficient virtual machine (VM) | interpreters from simple descriptions of the VM instructions. [ One of the nice/useful features of the the Forth VM is the PFA/CFA pairing: PFA is "parameter field address" and points to the code ( VM or native ) to be executed. CFA is "code field address" and points to the code to interpret what's in the parameter field. For threaded code, it points to the threaded code interpreter; for native code, it points to the PFA -- i.e. native code is 'self interpreting' . BUILDS/DOES in Forth creates a data type (BUILDS) and defines code to addess the data type (DOES) that is pointed to by the CFA -- an early but very primitive object-orientation, but with only one method (later Forth's added QUADS and other methods to have separate ACCESS/UPDATE (read/write) methods. ] Re: Other VM implementations: I'm not very familiar with the internals of Squeak, but I suspect that it's worth looking at. They are, in any case, interested in some of the same sort of things. ( There was a recent thread about MIT's StarLogo -- which was originally written for the Mac using (I think) Lisp, and then a portable version was done using Java, but they were disappointed in performance, and I think they are looking at using Squeak now. ) Scheme48 is probably considered the best portable byte-code Scheme implementation. ( Don't know anything about it's internals myself ) A lot of other people who have tried using the Java VM for other languages have had complaints about various things that are difficult or impossible. ( Scheme folks couldn't have full call/cc, and there were two different attempts to add generics to Java -- one involved adding special bytecode support, and the other (Pizza -- now GJ -- Generic Java) tried to stick with a standard Java VM. ) -- Steve majewski

On Mon, 30 Jul 2001, "Steven D. Majewski" <sdm7g@Virginia.EDU> wrote:
Scheme48 is probably considered the best portable byte-code Scheme implementation. ( Don't know anything about it's internals myself )
Last I heard (admittedly, >1 yr. ago), it didn't support 64 bit architectures. -- gpg --keyserver keyserver.pgp.com --recv-keys 46D01BD6 54C4E1FE Secure (inaccessible): 4BD1 7705 EEC0 260A 7F21 4817 C7FC A636 46D0 1BD6 Insecure (accessible): C5A5 A8FA CA39 AB03 10B8 F116 1713 1BCF 54C4 E1FE Learn Python! http://www.ibiblio.org/obp/thinkCSpy

"Steven D. Majewski" <sdm7g@Virginia.EDU>:
But then, I've always thought that one of the problems with trying to optimize Python was that the VM was too high level.
No, the problem is that Python is just too darn dynamic! This is a feature of the language, not just the VM. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+

I was thinking a little about a Python/Perl VM merge. One problem I imagine would be difficult to reconcile is the subtle difference in semantics of various basic types. Consider the various bits of Python's (proposed) number system that Perl might not have (or want): rationals, automatic promotion from machine ints to longs, complex numbers. These may not work well with Perl's semantics. What about exceptions? Do Python and Perl have similar notions of what exceptional conditions exist? Skip

Actually, this may not be as big a deal as I thought before. The PVM doesn't have a lot of knowledge about types built into its instruction set. It knows a bit about classes, lists, dicts, but not e.g. about ints and strings. The opcodes are mostly very abstract: BINARY_ADD etc. --Guido van Rossum (home page: http://www.python.org/~guido/)

Guido> The PVM doesn't have a lot of knowledge about types built into Guido> its instruction set.... The opcodes are mostly very abstract: Guido> BINARY_ADD etc. Yeah, but the runtime behind the virtual machine knows a hell of a lot about the types. A stream of opcodes doesn't mean anything without the semantics of the functions the interpreter loop calls to do its work. I thought the aim of Eric's Parrot idea was that Perl and Python might be able to share a virtual machine. If both can generate something like today's BINARY_ADD opcode, the underlying types of both Python and Perl better have the same semantics. Skip

Skip Montanaro wrote:
I don't think that needs to be true _in toto_. In other words, some opcodes can be used by both languages, some can be language-specific. The implementation of the VM for a given opcode can be shared per language, or even just partially shared. BINARY_ADD can do the same thing in most languages for 'native' types, and defer to per-language codepaths for objects, for example. One problem with a hybrid approach might be that optimizations become really hard to do if you can't assume much about the semantics, or if you can only assume the union of the various semantics. But the idea is intriguing anyway =). --david

Yeah, but the runtime could offer a choice of data types -- for Python code the constants table would contain Python ints and strings etc., for Perl code it would contain Perl string-number objects. Maybe. --Guido van Rossum (home page: http://www.python.org/~guido/)

Guido> Yeah, but the runtime could offer a choice of data types -- for Guido> Python code the constants table would contain Python ints and Guido> strings etc., for Perl code it would contain Perl string-number Guido> objects. Maybe. So I could give a code object generated by the Python compiler to the Perl runtime and get different results than if it was executed by the Python environment? Perhaps it's time for Eric to chime in again and tell us what he really has in mind. I can't see the utility in having the same set of opcodes for the two languages if the semantics of running them under either environment aren't going to be the same. It seems like it would artificially constrain people working on the internals of both languages. Skip

Skip Montanaro <skip@pobox.com>:
No, I don't think that's what Guido is saying. He and I are both imagining a *single* runtime, but with some type-specific opcodes that are generated only by Perl and some only generated by Python.
You're right. What I have in mind starts with a common opcode interpreter, perhaps based on the Python VM but with extended opcodes where Perl type semantics don't match, and a common callout mechanism to C-level runtime libraries linked to the opcode interpreter. In the conservative version of this vision, Perl and Python have different runtimes dynamically linked to an instance of the same opcode interpreter. Memory allocation/GC and scheduling/threading are handled inside the opcode interpreter but the OS and environment binding is (mostly) in the libraries. Things Python would bring to this party: our serious-cool GC, our C extension/embedding system (*much* nicer than XS). Things Perl would bring: blazingly fast regexps, taint, flexitypes, references. In the radical version, the Perl and Python runtimes merge and the differences in semantics are implemented by compiling different wrapper sequences of opcodes around the library callouts. At this point we're doing something competitive with Microsoft's CLR. My proposed work plan is: 1. Separate the Python VM from the Python compiler. Initially it's OK if they still communicate by hard linkage but that will change later. 2. Build the Parrot VM out from the Python VM by adding the minimum number of Perliferous opcodes. 3. Start building the Perl runtime on top of that, re-using as much of the Python runtime as possible to save effort. -- <a href="http://www.tuxedo.org/~esr/">Eric S. Raymond</a> Every election is a sort of advance auction sale of stolen goods. -- H.L. Mencken

[Eric, could you forward this to python-dev if it doesn't show of its own accord? I'm not yet subscribed, so I don't know if it'll make it] I should start with an apology for not being on python-dev when this started. Do please Cc me on anything, as I've not gotten on yet. (My subscription's caught in the mail, I guess... :) At 04:14 AM 7/31/2001 -0400, Eric S. Raymond wrote:
Odds are there won't even be a different set of opcodes. (Barring the possibility of the optimizer being able to *know* that an operation is guaranteed to be integer or float, and thus using special-purpose opcodes. And that's really an optimization, not a set of language-specific opcodes) The behaviour of data is governed by the data itself, so Python variables would have Python vtables attached to them guaranteeing Python behaviour, while perl ones would have perl vtables guaranteeing perl behaviour. This was covered, more or less, by the chunks of the internals talk I didn't get to. Slides, for the interested, are at http://dev.perl.org/perl6/talks/. I'm not sure if there's enough info on the slides themselves to be clear--they were written to be talked around.
I've snipped the rest here. I don't think Parrot will be built off the Python interpreter. This isn't out of any NIH feelings or anything--I'm obligated to make it work for Perl, as that's the primary point. If we can make Python a primary point too that's keen, and something I *want*, but I do need to keep focused on perl. Having said that, what I'm doing is stepping back from perl and trying, wherever possible, to make the runtime generic. If there's no reason to be perl specific I'm not, and so far that's not been a problem. (It actually makes life easier in a lot of ways, since we can then delegate the decision on how things are done to the variables involved, providing a default set of behaviours which the parser will end up determining anyway) On some things I think I'm being a bit more vicious than, say, Python is by default. (For example, if extension code wants to hold on to a variable across a GC boundary it had darned well better register that fact with the interpreter, or it's going to find itself with trash) I'm not sure about the extension mechanism in general--I've not had a chance to look too closely at what Python does now, but I don't doubt that, at the C level, the differences between the languages will be pretty trivial and easily abstractable. Seeing what you folks have is on the list 'o things to do--I may well steal from it wholesale. :) I expect there's a bunch of stuff I'm missing here, so if anyone wants to peg me with questions, go for it. (Cc me if they're going to the dev list please, at least until I'm sure I'm on) I really would like to see Parrot as a viable back end for Python--I think the joint development resources we could muster (possibly with the Ruby folks as well) could get us a VM for dynamically typed languages to rival the JVM/.NET for statically typed ones. Dan --------------------------------------"it's like this"------------------- Dan Sugalski even samurai dan@sidhe.org have teddy bears and even teddy bears get drunk

Hm, the Python bytecode interpreter doesn't do the memory allocation/GC, and only does the scheduling/threading indirectly (by releasing and immediately reacquiring the global interpreter lock).
I'm not sure I want references, and I don't know what flexitypes are. --Guido van Rossum (home page: http://www.python.org/~guido/)

At 07:15 PM 7/31/2001 -0400, Guido van Rossum wrote:
The Parrot interpreter does (or will at least, as there's not much code behind it yet) handle GC and intercept memory allocation, at least if you want to use managed memory. (Though that's only of interest to people who are writing C/C++/Fortran/Whatever extensions to the interpreter) GC & object destruction are going to be dealt with separately by the interpreter, FWIW. Which might not be much. As for threading, well, that's where things get interesting. Perl's tried it two ways (multiple threads in the same interpreter, and one thread per interpreter, with cloned interpreters) both of which aren't very good. And the global lock thing's not that keen either. The plan for parrot is to allow multiple interpreters to run simultaneously, with shared data protected on access time. (And *only* shared data protected, so there's no penalty to access non-shared data) Along with a scheme to allow only partial (or no) visibility of the spawning thread's variables. We plan on basing the secure sandbox scheme around this as well.
You certainly don't have to have them. The parser doesn't have to necessarily expose all the features of the interpreter. What you'd expose to the programmer would depend on the language grammar--if you can't write valid Python code to take references, well, you don't get them. :)
and I don't know what flexitypes are.
I presume this is perl's "I'll make it any type I darned well need" basic scalar, or possibly it's auto-resizing arrays and hashes. Neither of which you have to have either. Since that behaviour's a function of the type of a variable, we'd just not give python variables perl variable types. (Or the vtables that go with them) Now, what happens when a Python routine is passed a perl variable is an interesting question. Personally I'd presume that it Did The Right Thing, but I'm not 100% sure what that is. Probably maintain its core behavior. (So if I passed in a Magic Morphing Perl Scalar to a Python routine that needed an integer value, it'd get one, but if I tried to use a Python integer in string context in a perl sub, I'd get yelled at. (Assuming that's the correct behaviour, of course. If not, well, it'd do something else)) Dan --------------------------------------"it's like this"------------------- Dan Sugalski even samurai dan@sidhe.org have teddy bears and even teddy bears get drunk

Dan Sugalski wrote:
What is the downside of the global lock on the average single processor machine? I tend to think that the "default" threading model should allow simple and easy, everything-shared multi-threading on ordinary machines. Having a multi-processor-friendly advanced mode is a great extension for the wizards. -- Take a recipe. Leave a recipe. Python Cookbook! http://www.ActiveState.com/pythoncookbook

At 04:42 PM 7/31/2001 -0700, Paul Prescod wrote:
If you hold the lock during an I/O operation, you'll lose time you could have otherwise used. Getting and releasing a global lock frequently also costs performance you might otherwise have used in other places. Mutex releases require memory coherency, which will force your CPU to flush any pending writes that might be hanging about, which will tend to drop it's efficiency, especially on heavily out-of-order machines like the Alpha. Also, that is a zillion and a half mutex aquisition and releases, most of which you probably have no need of. Even cutting out, say 5-10% of them will be noticeable. (I do work on SMP machines as a rule, so I am a little biased against things that single-thread me when I don't need it--what's the point of 500% idle time?) Dan --------------------------------------"it's like this"------------------- Dan Sugalski even samurai dan@sidhe.org have teddy bears and even teddy bears get drunk

To bring in yet someone else to this discussion, Mark Hammond and I were chatting with Brendan Eich (of JavaScript and Mozilla fame) last week about interpreter locks and the like, and Brendan mentioned that he had done a lot of possibly reusable work in optimizing the various kinds of thread lock strategies in JavaScript, as JS is used a lot server-side. Specifically he was talking about some 'multi-level' locking stuff which sounded impressive to me at least. I doubt if Brendan has time to dig into either of Perl or Python's internals, but I'm sure he'll provide general pointers if folks ask nicely. =) Brendan, do you care to give an executive overview of the locking strategies used in JS? --david ascher PS: Brendan: If you want to know the context of this discussion, see: http://aspn.activestate.com/ASPN/Mail/Browse/Threaded/python-dev

BTW, for a VM flexible enough to deal with Python and Perl, JavaScript should be a piece of cake. (JavaScript is much closer to Python than to Perl in its semantic model, for the most part.) --Guido van Rossum (home page: http://www.python.org/~guido/)

Dan Sugalski wrote:
"Doctor it hurts when I do that." :) We just don't hold the lock during I/O or other computations that do not affect Python interpreter data structures. -- Take a recipe. Leave a recipe. Python Cookbook! http://www.ActiveState.com/pythoncookbook

Dan Sugalski wrote:
If you hold the lock during an I/O operation, you'll lose time you could have otherwise used.
Python extensions can (and do, for all standard I/O operations) release the Global Interpreter Lock. For more information, check out http://starship.python.net/crew/aahz/ -- --- Aahz (@pobox.com) Hugs and backrubs -- I break Rule 6 <*> http://www.rahul.net/aahz/ Androgynous poly kinky vanilla queer het Pythonista I don't really mind a person having the last whine, but I do mind someone else having the last self-righteous whine.

[Paul Prescod]
[Dan Sugalski]
Python doesn't actually suffer from either of these problems: while there's a pair of acquire/release-global-lock macros around potentially blocking I/O calls in Python's runtime (ditto sleep(), etc), no mutex is actually allocated before *somebody* calls PyEval_InitThreads: void PyEval_InitThreads(void) { if (interpreter_lock) return; _PyThread_Started = 1; interpreter_lock = PyThread_allocate_lock(); PyThread_acquire_lock(interpreter_lock, 1); main_thread = PyThread_get_thread_ident(); } In most uses of Python, no thread other than the main thread ever gets created, that routine never gets called, interpreter_lock remains NULL, and all the global-lock acquire/release code reduces to a cheap test against NULL. However, Python calls the platform's thread-safe libraries regardless, and *that* can be a huge speed hit. A minor example is that system malloc() is more expensive in Microsoft's thread-safe version of libc. A monster example is speed of line-at-a-time input: we only recently discovered that Python's getc()-in-a-loop was killing us on many platforms because the platform threadsafe library implementation locked and unlocked the stream for each character. Worming around that brought our input speed much closer to Perl's (up to 50x faster on Tru64 Unix). It's still slower on most boxes, though, because we're still threadsafe, but, last I looked, Perl's line-at-a-time input tricks mucked with stdio structs directly without benefit of exclusion (and are not threadsafe).
Greg Stein is the fellow to talk with about about "free threading" of Python. He had that at least mostly working several years ago, but it was a major project, that patch is way out of date now, and Python is much more elegant now. Oops! I didn't mean "elegant", I meant "bigger" <wink>.

At 01:23 AM 8/1/2001 -0400, Tim Peters wrote:
Yeah, I figured you didn't initialize or aquire the mutex until it was actually needed. One of the nice side-benefits of the global opcode-aquired lock.
Everywhere else too, I'd bet. I've been considering thread-specific memory pools because of this. (Well, this is one reason, at least)
Ouch, I'd bet that hurts. Has anyone timed the difference between making lots of getc calls and making a few larger reads and managing the buffers internally? I can see it going either way, and another data point would be useful to have.
Worming around that brought our input speed much closer to Perl's (up to 50x faster on Tru64 Unix).
FWIW, at least on Tru64 and VMS, there are a number of faster thread calls if you don't mind bypassing some library error checking. (Which is OK if you're guaranteed to have correct parameters) I can dig them up if you like. A tweak of pthread_get_specific made a depressingly large performance boost for me on my VMS box. (I did say perl's threading models weren't that good... :) Not worth it if you don't make many calls, though.
Yep, perl's pthread-based threading model doesn't guarantee threadsafe I/O on many platforms. The alternative threading model doesn't use that codepath. (It's currently primarily windows-based, and we don't do the buffer lookbehind stuff there)
:) I'm as much (or more, which is generally an odd thing, but threads are almost inherently odd) interested in how things look at the programmer level and what guarantees are made as how things look under the hood. (It's all just a SMOP, right?) Dan --------------------------------------"it's like this"------------------- Dan Sugalski even samurai dan@sidhe.org have teddy bears and even teddy bears get drunk

[Dan Sugalski, on Wednesday, August 01, 2001 2:20 AM]
There's been lots of this in Python-Dev, like in this thread: http://aspn.activestate.com/ASPN/Mail/Message/600485 I'll quote the high-order bit: My line-at-a-time test case used (rounding to nearest whole integers) 30 seconds in Python and 6 in Perl. The result of testing many changes to Python's implementation was that the excess 24 seconds broke down like so: 17 spent inside internal MS threadsafe getc() lock/unlock routines 5 uncertain, but evidence suggests much of it due to MS malloc/realloc (Perl does its own memory mgmt) 2 for not copying directly out of the platform FILE* implementation struct in a highly optimized loop (like Perl does) My last checkin to fileobject.c reclaimed 17 seconds on Win98SE while remaining threadsafe, via a combination of locking per line instead of per character, and invoking realloc much less often (only for lines exceeding 200 chars). Note that thread overhead is overwhelmingly the biggest hangup. Python has two threadsafe input tricks now: 1. On platforms that have flockfile(), funlockfile(), and getc_unlocked(), the last is used in a loop bracketed by the first two. 2. At least on Windows, which doesn't have those, we use the platform fgets() in an excruciating way, tricking it into letting us read lines with embedded null bytes. Oddly enough, in the timing reports I saw, approach #1 was never faster than approach #2, and on at least one platform (Tru64, IIRC) was slower. Of course fgets() is a primitive in std C because they *wanted* to make it possible for vendors to optimize it (in the ways Perl does), but it appears very few vendors do optimize it. On Windows it's the same old getc()-in-a-loop, but they lock/unlock the stream only once per fgets call (using internal stream functions that aren't exposed). The "2 seconds for not copying directly ... like Perl does" I reported above came from hacking together a thread-unsafe line input routine that used the same FILE* tricks Perl uses. That is, thread-unsafe getc-in-a-loop was 2 seconds slower than using thread-unsafe FILE* tricks. That's significant in absolute terms, but was lost in the noise compared to the other stuff we were fighting.
participants (26)
-
aahz@rahul.net
-
Andrew Kuchling
-
Dan Sugalski
-
David Ascher
-
Eric S. Raymond
-
Gavin Sherry
-
Gordon McMillan
-
Greg Ewing
-
Greg Ward
-
Guido van Rossum
-
Guido van Rossum
-
Jeremy Hylton
-
Ka-Ping Yee
-
Moshe Zadka
-
Nathan Torkington
-
Neil Schemenauer
-
Paul Prescod
-
Rasmus Lerdorf
-
Samuele Pedroni
-
Simon Cozens
-
Skip Montanaro
-
Sterling Hughes
-
Steven D. Majewski
-
Thomas Wouters
-
Tim Peters
-
Tim Peters