MOPs (warning: LONG)

[Joachim Durchholz]

Pascal Costanza wrote:

> Joachim Durchholz wrote:
> 
>> Pascal Costanza wrote:
>>
>>> For example, static type systems are incompatible with dynamic 
>>> metaprogramming. This is objectively a reduction of expressive power, 
>>> because programs that don't allow for dynamic metaprogramming can't 
>>> be extended in certain ways at runtime, by definition.
>>
>> What is dynamic metaprogramming?
> 
> Writing programs that inspect and change themselves at runtime.

That's just the first part of the answer, so I have to make the second 
part of the question explicit:

What is dynamic metaprogramming good for?

I looked into the papers that you gave the URLs on later, but I'm still 
missing a compelling reason to use MOP. As far as I can see from the 
papers, MOP is a bit like pointers: very powerful, very dangerous, and 
it's difficult to envision a system that does the same without the power 
and danger but such systems do indeed exist.


(For a summary, scroll to the end of this post.)


Just to enumerate the possibilities in the various URLs given:

- Prioritized forwarding to components
(I think that's a non-recommended technique, as it either makes the 
compound object highly dependent on the details of its constituents, 
particularly if a message is understood by many contituents - but 
anyway, here goes:) Any language that has good support for higher-order 
functions can to this directly.

- Dynamic fields
Frankly, I don't understand why on earth one would want to have objects 
with a variant set of fields. I could do the same easily by adding a 
dictionary to the objects, and be done with it (and get the additional 
benefit that the dictionary entries will never collide with a field name).
Conflating the name spaces of field names and dictionary keys might 
offer some syntactic advantages (callers don't need to differentiate 
between static and dynamic fields), but I fail to imagine any good use 
for this all... (which may, of course, be lack of imagination on my 
side, so I'd be happy to see anybody explain a scenario that needs 
exactly this - and then I'll try to see how this can be done without MOP 
*g*).

- Dynamic protection (based on sender's class/type)
This is a special case of "multiple views" (implement protection by 
handing out a view with a restricted subset of functions to those 
classes - other research areas have called this "capability-based 
programming").

- Multiple views
Again, in a language with proper handling for higher-order functions 
(HOFs), this is easy: a view is just a record of accessor functions, and 
a hidden reference to the record for which the view holds. (If you 
really need that.)
Note that in a language with good HOF support, calls that go through 
such records are syntactically indistinguishable from normal function 
calls. (Such languages do exist; I know for sure that this works with 
Haskell.)

- Protocol matching
I simply don't understand what's the point with this: yes of course this 
can be done using MOP, but where's the problem that's being simplified 
with that approach?

- Collection of performance data
That's nonportable anyway, so it can be built right into the runtime, 
and with less gotchas (if measurement mechanisms are integrated into the 
runtime, they will rather break than produce bogus data - and I prefer a 
broken instrument to one that will silently give me nonsense readings, 
thank you).

- Result caching
Languages with good HOF support usually have a "memo" or "memoize" 
function that does exactly this.

- Coercion
Well, of all things, this really doesn't need MOP to work well.

- Persistency
(and, as the original author forgot: network proxies - the issues are 
similar)
Now here's a thing that indeed cannot be retrofitted to a language 
without MOP.
(Well, performance counting can't be retrofitted as well, but that's 
just a programmer's tool that I'd /expect/ to be part of the development 
system. I have no qualms about MOP in the developer system, but IMHO it 
should not be part of production code, and persistence and proxying for 
remote objects are needed for running productive systems.)


For the first paper, this leaves me with a single valid application for 
a MOP. At which point I can say that I can require that "any decent 
language should have this built in": not in the sense that every 
run-time system should include a working TCP/IP stack, but that every 
run-time system should include mechanisms for marshalling and 
unmarshalling objects (and quite many do).


On to the second paper (Brant/Foote/Johnson/Roberts).

- Image stripping
I.e. finding out which functions might be called by a given application.
While this isn't Smalltalk-specific, it's specific to dynamic languages, 
so this doesn't count: finding the set of called functions is /trivial/ 
in a static language, since statically-typed languages don't usually 
offer ways to construct function calls from lexical elements as typical 
dynamic languages do.

- Class collaboration, interaction diagrams
Useful and interesting tools.
Of course, if the compiler is properly modularized, it's easy to write 
them based on the string representation, instead of using reflective 
capabilities.

- Synchronized methods, pre/postcondition checking
Here, the sole advantage of having an implementation in source code 
instead of in the run-time system seems to be that no recompilation is 
necessary if one wishes to change the status (method is synchronized or 
not, assertions are checked or not).
Interestingly, this is not a difference between MOP and no MOP, it's a 
difference between static and dynamic languages.
Even that isn't too interesting. For example, I have worked with Eiffel 
compilers, and at least two of them do not require any recompilation if 
you want to enable or disable assertion checking (plus, at least for one 
compiler, it's possible to switch checking on and off on a per-program, 
per-class, or even per-function basis), so this isn't the exclusive 
domain of dynamic languages.
Of course, such things are easier to add as an afterthought if the 
system is dynamic and such changes can be done with user code - but 
since language and run-time system design are as much about giving power 
as guarantees to the developer, and giving guarantees necessarily 
entails restricting what a developer can do, I'm entirely unconvinced 
that a dynamic language is the better way to do that.

- Multimethods
Well, I don't see much value in them anyway...


... On to Andreas Paepcke's paper.
I found it more interesting than the other two because it clearly spells 
out what MOPs are intended to be good for.

One of the main purposes, in Paepcke's view, is making it easier to 
write tools. In fact reflective systems make this easier, because all 
the tricky details of converting source code into an internal data 
object have already been handled by the compiler.
On the other hand, I don't quite see why this should be more difficult 
for a static language.
Of course, if the language designer "just wanted to get it to compile", 
anybody who wants to write tools for the language has to rewrite the 
parser and decorator, simply because the original tools are not built 
for separating these phases (to phrase it in a polite manner). However, 
in the languages where it's easy to "get it to compile" without 
compromising modularity, I have seen lots of user-written tools, too. I 
think the main difference is that when designing a run-time system for 
introspection, designers are forced to do a very modular compiler design 
- which is a Good Thing, but you can do a good design for a 
non-introspective language just as well :-)

In other words, I don't think that writing tools provides enough reason 
for introspection: the goals can be attained in other ways, too.


The other main purpose in his book is the ability to /extend/ the 
language (and, as should go without saying, without affecting code that 
doesn't use the extensions).
He claims it's good for experimentation (to which I agree, but I 
wouldn't want or need code for language experimentation in production code).

Oh, I see that's already enough of reasons by his book... not by mine.



Summary:
========

Most reasons given for the usefulness of a MOP are irrelevant. The 
categories here are (in no particular order):
* Unneeded in a language without introspection (the argument becomes 
circular)
* Easily replaced by good higher-order function support
* Programmer tools (dynamic languages tend to be better here, but that's 
more of a historical accident: languages with a MOP are usually highly 
dynamic, so a good compiler interface is a must - but nothing prevents 
the designers of static languages from building their compilers with a 
good interface, and in fact some static languages have rich tool 
cultures just like the dynamic ones)

A few points have remained open, either because I misunderstood what the 
respective author meant, or because I don't see any problem in handling 
the issues statically, or because I don't see any useful application of 
the mechanism. The uses include:
* Dynamic fields
* Protocol matching
* Coercion

And, finally, there's the list of things that can be done using MOP, but 
where I think that they are better handled as part of the run-time system:
* (Un-)Marshalling
* Synchronization
* Multimethods

For (un-)marshalling, I think that this should be closed off and hidden 
from the programmer's powers because it opens up all the implementation 
details of all the objects. Anybody inspecting source code will have to 
check the entire sources to be sure that a private field in a record is 
truly private, and not accessed via the mechanisms that make user-level 
implementation of (un-)marshalling possible.
Actually, all you need is a builtin pair of functions that convert some 
data object from and to a byte stream; user-level code can then still 
implement all the networking protocol layers, connection semantics etc.

For synchronization, guarantees are more important than flexibility. To 
be sure that a system has no race conditions, I must be sure that the 
locking mechanism in place (whatever it is) will work across all 
modules, regardless of author. Making libraries interoperate that use 
different locking strategies sounds like a nightmare to me - and if 
everybody must use the same locking strategy, it should be part of the 
language, not part of a user-written MOP library.
However, that's just a preliminary view; I'd be interested in hearing 
reports from people who actually encountered such a situation (I 
haven't, so I may be seeing problems where there aren't any).

For multimethods, I don't see that they should be part of a language 
anyway - but that's a discussion for another thread that I don't wish to 
repeat now (and this post is too long already).


Rambling mode OFF.

Regards,
Jo