The Design of a Metaobject Protocol Controlling Behavior of a

The Design of a Metaobject Protocol Controlling Behavior of a Scheme Interpreter Amin Vahdat March 10, 1993

1 Introduction Recent work strongly suggests people prefer to work with systems which are open, that is, which allow their user to tailor system behavior according to their individual needs. This notion of open behavior has recently been shown to be applicable to programming languages in [KdRB91, Rod91] through the use of metaobject protocols. Eorts in this area have been largely focused on increasing the performance of compilers. However, these same ideas can be used to give the user control over the actual semantics of the programming language. In order to experiment with the utility of this kind of ability, we have designed a metaobject protocol for an interpreter1 with the base semantics of Scheme [Cli91]. Through the MOP, we provide a protocolization of the essential structure of a language implementation designed in a way to expose those aspects of the interpreter's behavior which we would like the user to access and control. In this paper, we describe a short experiment which begins to evaluate the utility of user controlled semantics. We discuss the design of this metaobject protocol to demonstrate some of our protocol design methodology and then give some preliminary examples showing the utility of user controlled semantics. In conclusion, we present a brief overview of all the working semantic extensions and sketch some areas for future work.

2 The Notion of Flexible Semantics The motivation for our experiment is based on the intuition that the semantics of a programming language is suited to a xed domain of computational problems. Thus, under a xed set of semantics, there will always be some computations which will be dicult to express. If, on the other hand, the language's semantics were more exible, then the programmer could choose the set of semantic rules which would most facilitate the expression of his ideas. We have designed a programming language with exible semantics in order to focus on increasing the programmer's ability to clearly and concisely express his ideas. There are at least two approaches to developing a language with exible semantics. One method is to provide a xed set of \canned" semantic rules which the programmer could choose from. This Readers not familiar with metaobject protocols should refer to appendix A for a brief discussion of the application of a metaobject protocol to an interpreter. 1

1

method, however, suers from the same problem that a language with a xed set of semantics suers from: that some computations will not be easily or naturally expressible by any of the dierent semantics choices provided to the programmer. Another method is to develop a paradigm which allows the programmer to extend the set of semantic rules of the programming language in a principled fashion. By principled, we mean that the facility provided to allow user extensions to the semantics must have the following properties:  incrementality - If the programmer wishes to change a certain aspect of the semantics of the language, then it should not be necessary to have global knowledge of the details behind other semantic aspects of the language.  locality - The programmer should have the ability to restrict semantic changes to appropriate areas within the program. Flexible semantics with the above freedoms can greatly facilitate the programmer's ability to express certain computations. Furthermore, since these changes to the semantics are local, the programmer is free to alter semantics to make parts of the program easier to express. Previous MOPs opened up compiler abstraction (implementation issues) allowing the programmers to become compiler designers when the need arose. When decisions made in the implementation of a compiler adversely aects user programs, the programmer has the ability to modify relevant compiler decisions. Our experiment draws a parallel to programming language design: when semantics of a programming language aect the programmer's ability to express his work, the programmer can, in eect, locally redesign the programming language. The hope is to enhance expressibility rather than performance.

3 The Development of Ploy Faced with the problem of programming languages with xed semantics which can only be applicable to a nite number of users, we hypothesize that a language with semantics open to user control will not only appeal to a wider range of users, but will also allow these users to express their computations in a much more succinct and clear fashion. In order to experiment with this theory of user-controlled semantics, we have built a prototype interpreter called Ploy whose base semantics are those of Scheme. Our primary goal in designing Ploy was to provide an interpreter which would allow users to implement a number of semantic extensions, such as dynamic scoping or normal order evaluation, in as simple and concise a manner as possible. Ploy is designed to allow its users to design new programming languages as demonstrated by the following general scenario:  In writing a piece of code, the programmer decides that some (non-default) semantics would more conveniently express his computation.  He then decides on the form which these new semantics should take in the program and decides on the aspects of the programming language which he needs to control to achieve this functionality.  After designing the speci cations for this new language and outlining the requirements to make the semantic change, the programmer makes use of Ploy's metaobject protocol to implement the desired changes. 2

Thus, the metaobject protocol designer's role in this scenario is to provide logical and clean access points for a wide variety of possible semantic extensions (an interface for the language designer).

4 Derivation of Ploy's Protocol Ploy's basic structure closely resembles that of Anibus [Rod91]. User programs are parsed into a program graph re ecting their logical structure. Nodes within the program graph are metaobjects. Evaluation of these metaobjects is done through methods which specialize on one or metaobjects. Users can specialize the behavior of the language through marks on the program source. As will be demonstrated below, these marks on the program source specify the instantiation of non-default metaobjects in the program graph. The user can then write new methods which specify on the marked metaobject to locally specialize language semantics. Thus, the types of metaobjects provided by the protocol would determine the kind of control the programmer has over the language semantics. We chose our metaobjects to correspond to the three separate, but inter-dependent constructs described for the scheme interpreter in [ASS85]. First, there must be a representation of the program to be interpreted. Second, the interpreter must maintain an internal state to determine the context under which a statement is to be evaluated. Finally, the interpreter must be able to represent the entire range of possible return values and communicate these values to the user or to other parts of the program. The three access points for user-implemented semantic modi cation are then:

 program element - These kinds of metaobjects represent the structure of the Scheme source

code to be interpreted and executed. Such metaobjects may represent constructs such as let, lambda, or set!.  interpreter state - Ploy's state contains these kinds of metaobjects which in turn represent the environment in which program elements are executed.  scheme value - These kinds of metaobjects represent the values returned upon evaluation of program elements; they further represent the range and domain of all Scheme expressions.

With control over these metaobjects, the programmer has broad in uence over the various aspects of the interpreter's work. In the base case (corresponding to no modi cations by the programmer), external and internal constructs in Ploy correspond to the leaves in the default class inheritance tree shown in Figure 1. The next task in designing Ploy's protocol is to determine the tasks of the generic functions which control the interaction of Ploy's three kinds of metaobjects. We believe that the generic functions in Ploy's protocol provide a simple, clean, and powerful interface to many important aspects of language semantics. In order to elucidate some of the decisions which contributed to the nal protocol, we rst describe in detail the development of two sample semantic extensions to Ploy.

4.1 Monitored Variables

The rst example we present is monitored variables. When debugging a program, it is often very convenient to print a message each time a variable is read or set. In large programs however, it can 3

metaobject

program-element

runtime-data

.... let

procedure-call interpreter-state

scheme-value ....

binding

pair

procedure

primop

closure

Figure 1: An overview of the default classes in Ploy, showing the three basic categories of metaobject: program-element, interpreter-state, and scheme-value. often be very tedious to manually insert debugging output before every reference to a variable in question. It would be much more convenient to specify certain variables to be of a special monitored variety. That is, variables declared to be monitored will always print a message upon being set or read2 . If we are to introduce monitored variables into Ploy, we must provide a method to distinguish between them and normal, non-monitored variables. Clearly, if a semantic change to Ploy caused all variables to become monitored, an inordinate amount of irrelevant information would be provided to the user. To facilitate the coexistence of the two dierent types of variables in Ploy, we use marks enclosed within curly braces on the base level program to declare certain Scheme constructs (in this case variables) to behave dierently from the default. In this example, marks are used on the source program to specify that a particular variable is to be of the marked variety. Note that marks are not intrinsic to metaobject protocols; they are simply an enabling technology allowing constructs with dierent semantic paradigms to conveniently coexist in the same program. Given the ability to distinguish between the default and user-de ned constructs, the programmer can now decide on the speci c behavior of monitored variables. One possibility for such behavior is illustrated in the following sample session with Ploy: PLOY ==> (define {monitored-variable}x 0) x PLOY ==> (set! x 7)

Note that this example of monitored variables is just a special case of the more general notion of active variables. An active variable runs a user de ned function whenever it is set, read, or bound. The extensions necessary to implement active variables once monitored variables have been implemented are straight forward. 2

4

Setting the value of x to 7. 7 PLOY ==> (define y (+ x 1)) Reading the value of x. y

Upon deciding on the speci c behavior and form which the new semantics is to take, the programmer must next decide which aspects of the interpreter must be modi ed to achieve the new behavior. In the case of monitored variables, the programmer must have access to the point where variable values are set and read. In a Scheme interpreter, variable values are accessed through bindings in the environment; there is a one to one correspondence between de ned variables in the source program and bindings in the environment. As described above, the programmer has control over the environment through the binding metaobject (there is also one instance of this metaobject for each binding in the environment). Given the fact that control over the manipulation of metaobjects is achieved through generic functions and also given the fact that the user needs control over a given aspect of Ploy's functionality to implement monitored variables, we must endeavor to design the protocol with generic functions which provide control over the relevant interpreter functionality. To this end, Ploy's protocol provides two generic functions get-value and set-value that specialize on instances of the binding metaobject. They are called whenever a variable value is accessed by a program. Thus, by creating a class which inherits from the default binding, the user can write methods on get-value and set-value to print out messages whenever variables corresponding to a special class of binding are accessed. The following meta-code illustrates this process3 : (defclass monitored-binding (binding) ()) (defmethod get-value :before ((b monitored-binding)) (format t "Reading the value of ~S." b) (defmethod set-value :before ((b monitored-binding) new) (format t "Setting the value of ~S to ~S." b new))

However, these two methods dispatch on bindings, rather than variables, which is the metaobject class which the user has direct control over through marks on the base-level program. however, there is a direct correspondence between instances of binding and instances of variable, and, in a sense, every variable in the source program gives rise to an instance of a binding. Given this intuition that users must be able to specialize on Ploy's internal metaobjects despite the fact that there is no direct way to mark them in the text of the program, we, as the protocol designers, must provide some way of giving the user causal reach over internal metaobjects. This facility is provided through generic functions specializing on instances of program-element which are responsible for instantiating Ploy's internal metaobjects. In this case, where the user needs to specialize on bindings, the generic function make-binding is the relevant access point. This method specializes on variables; thus a method on make-binding Note that this code is written in CLOS, rather than an object oriented extension of Scheme. This was done partially to allow for easily distinguishing between base-level and meta-level code. 3

5

which dispatches on instances of monitored-variable can be responsible for creating a monitoredbinding in the following way: (defclass monitored-variable (variable) ()) (defmethod make-binding ((variable monitored-variable) value) (let ((result (call-next-method))) (change-class result 'monitored-binding) result))

This example demonstrates the general methodology for extending Ploy's semantics. Marks are made on the base program indicating that speci c constructs should be of a user-de ned class. Next, at the points in the protocol where behavior must be changed, methods are de ned which dispatch on these new classes. At times (as was the case with get-value and set-value in the monitored variables example above), generic functions are not called with classes corresponding to constructs in the base program, but with metaobject classes internal to Ploy. In these cases, other generic functions must be used to instantiate these internal metaobjects based on some base program construct. The monitored variables example also demonstrates some important considerations in Ploy's metaobject protocol design. In implementing various semantic extensions to Scheme, we attempted to decide which interpreter computations and actions the user would need to control. Through the provision of generic functions with strictly de ned invariants and functionality, we attempted to split these computations with enough granularity so that only a small number of methods need be written to implement the desired semantic changes. Ploy's structure thus satis es our notion of incrementality by allowing the user to simply \plug in" methods which implement the necessary semantic changes; knowledge of a large number of the generic functions, their methods, and metaobjects is not necessary to implement any given semantic extension. With the monitored variables example, the user did not need knowledge of how bindings are used in the rest of the interpreter, how they relate to other metaobjects, or how they are actually created; he just needed to know which generic functions (or point of access) were responsible for creating and accessing the bindings.

4.2 Normal Order Evaluation

To demonstrate user specialization on procedure call semantics, we demonstrate how the programmer can locally change a procedure call's argument evaluation from applicative order (the default in Scheme) to normal order. With applicative order evaluation, all arguments to a procedure are evaluated before the actual call is made. With normal order evaluation however, arguments are not evaluated until they are absolutely needed for some other computation (application to a strict operator such as addition, evaluation of a conditional expression, or application of a procedure). Thus, with this functionality in mind, the programmer may decide on the following dierentiation between applicative and normal order procedure calls: (define (try a b) (if (= a 0) 1

(define (try a {delayed-var}b) (if (= a 0) 1

6

b)))

b))

(try 0 (/ 1 0))

Invocation of the code on the left, which makes use of the default applicative order evaluation rules, signals a division by zero error. Invocation of the code on the right, which through marks on its formal arguments makes use of normal order evaluation rules, completes successfully and returns 1. To implement normal order evaluation, the programmer needs control over the point where arguments to a procedure are evaluated and also the point where arguments to a strict operator are forced. This functionality is provided with two generic functions, eval-arg and force. eval-arg is called by the interpreter whenever it needs to evaluate individual arguments to a procedure, and force is called on each evaluated argument to a strict operator before its arguments are actually applied. The default method on eval-arg evaluates the argument in the current environment, while the default method on force certainly returns the value passed to it. The necessary methods on these generic functions plus the de nition of the necessary metaobject classes follows: (defclass delayed-var (variable-binding-node) ()) (defclass thunk (scheme-value) ((expr :initarg :expr :accessor thunk-expr) (env :initarg :env :accessor thunk-env))) (defmethod eval-arg ((formal delayed-var) arg env) (make-instance 'thunk :expr arg :env env))) (defmethod force ((val thunk)) (force (eval (thunk-expr val) (thunk-env val))))

A thunk is a subclass of scheme-value which contains enough state information about an argument expression to allow the interpreter to evaluate it at a later time. The delayed-var class is necessary to mark the actual text of the program. With these de nitions, whenever an argument whose corresponding formal is an instance of delayed-var, a thunk is created which creates enough state information to evaluate the argument at a later point. At the points in the interpreter where expressions need to be forced (outline above), if force is called with a thunk, then the thunk's expression is evaluated recursively until a non-thunk is returned.

4.3 Ploy's Generic Functions

As demonstrated by the above examples, Ploy's generic functions logically partition the task of evaluation by dividing into layers which carry out progressively more speci c duties [KL92]. The layering of the protocol was largely driven by the desire to provide the most convenient access points for all of the semantic extensions which we implemented, rather than being tailored for ease of use with any single example. This layering can be seen in Figure 2. 7

Most Specific eval-arg eval-args getvalue

setvalue

makebinding

apply

return-value

force

eval

Figure 2: The layering of the generic functions in Ploy, showing the progression from the most general (bottom layer) to the most speci c (top layer) generic function. The lowest layer of functionality is eval which is called with a program-element and an environment which represents state. Depending on the class of the program-element to be evaluated, eval may then in turn call other, more speci c generic functions within the layered protocol, nally returning an instance of a subclass of scheme-value representing the evaluation of the expression. For example, if eval is called with a variable, eval will nd the binding matching the variable it was passed in its current environment and then call the generic function get-value with the binding to retrieve the scheme-value metaobject containing the value of the variable in question. The layering of the protocol also works to satisfy our requirement of incrementality in the following way. In the lowest layer, the user has the greatest amount of semantic control, but he also needs a fairly global knowledge of the workings of the interpreter to implement desired extensions. In the higher layers, the user has progressively decreasing amounts of power, but the prerequisite knowledge becomes more and more localized. For example, with monitored variables, the user does not have to know how variable values are accessed, just which generic functions are called to accomplish the task.

5 Issues/Results We have used Ploy to implement a fairly large number of semantic extensions (the code for these extensions and sample usage can be found in the appendix) to Scheme including:

 order of argument evaluation - Allow the arguments to a procedure to be evaluated in an

arbitrary order. Programmer needs control over argument evaluation (eval, eval-args).  dynamic-let - Allows for certain variables to be evaluated dynamically (based on callers frame) rather than the statically (eval).  partial closures - New semantic construction from the University of Indiana. Programmer must control evaluation of language constructs and must have knowledge of the maintenance of the environment (eval). 8

 active variables - Allow a user de ned procedure to be invoked whenever a variable is touched.    

Programmer needs to control creation of bindings and evaluation of variables (make-binding, get-value, set-value). active values - Allow a user de ned procedure to be invoked whenever a value is touched (values can migrate from variable to variable). Programmer must have control over expression evaluation and over metaobjects representing evaluated expressions (return-value, eval). dependent values - Allows for the declaration that a given variable is dependent on the value of a number of other values. The marked variables variable is automatically updated whenever the value of variables it is dependent upon changes (spreadsheet example). Programmer must control evaluation of variables and creation of bindings (eval, make-binding). call by need - Or applicative order evaluation; do not evaluate a procedure's actual arguments until the value is absolutely needed (as in a mathematical operation or a boolean check). Programmer needs access over argument evaluation of both primitive operators and user de ned procedures (eval, eval-arg). memoization - Cache the results of computationally intensive procedure calls. Programmer needs control over evaluation of procedure calls (apply, eval).

All the dierent semantic modi cations have ful lled our requirements for incrementality and scope control as evidenced by the compactness and relative clarity of the necessary code. None of the examples we have implemented has required more than 30 lines of code, and the entire protocol consists of nine generic functions, plus four other functions, thus satisfying our notion of incrementality. Furthermore, all semantic extensions satisfy our notion of scope control since the semantic changes are activated through marks on the base program. All our examples can be loaded simultaneously with dierent semantic extensions restricted to relevant code segments. Two key points came out of the development of the Ploy protocol. First, the fact that a large number of semantic extensions were implemented with a small protocol is quite signi cant. This means that prospective users of the Ploy MOP will not have to spend a large amount of time learning the protocol. Users will note have to write a lot of code to implement semantic extensions. This will encourage experimentation on the part of users, allowing them to determine which semantic rules they like most. The second important issue was the use of marks. Marks are an enabling technology allowing users to specialize the behavior of speci f portions of the program. However, the mark-transfer problem |the problem of being forced to mark program constructs (the externals of the program) in order to specialize the interpreters internals| is a general one. Such specialization is not completely natural for the user; ongoing work in programming languages at PARC endeavors to design a more dynamic object-oriented programming language which will address problems such as the mark transfer problem. Of course, our results are preliminary and there are many more semantic extensions which one could imagine adding to the protocol. One example of such an extension is an explicit control interpreter where the user has even more control over the interpreter's state. Our early success along with the amount of work left to do in the area suggest that further work in the area may prove fruitful in providing a number of compelling examples for the utility of open (or exible) semantics in programming languages. 9

Appendix A { An Interpreter's Metaobject Protocol As applied to an interpreter, a metaobject protocol consists of a number of metaobjects representing data structures containing information about the source program. Evaluation of the source program is achieved through generic functions which are called with the metaobjects representing the program. These generic functions may then modify or create other metaobjects containing the interpreter's state information. The user can change the behavior of the interpreter by creating new classes which inherit from the default metaobjects. Methods on relevant generic functions are then written which maintain a documented invariance, but which modify the default behavior in a desired way. Thus, the principle task in designing a metaobject protocol is deciding upon the constructs to be represented as metaobjects and then determining how these metaobjects interact through generic functions.

Appendix B { Ploy's Generic Functions Ploy makes use of a layered protocol of generic functions to carry out the task of interpretation. Two auxilary functions in the protocol cannot be customized by the programmer. However their use is important to the default implementation of the generic fuctions detailed above. In this appendix, we present complete speci cations for all generic and documented functions used by the Ploy interpreter.

scheme-eval node env This generic function takes a program-graph object, possibly modi es the interpreter's runtime state (which is entirely captured in the environment), and returns an instance of scheme-value (which represents the runtime values). Depending on the class of node, scheme-eval may call other generic functions within the layered protocol.

scheme-apply node procedure args env This generic function is called by scheme-eval in order to calculate the value returned by the application of a given list of arguments to a previously de ned procedure. scheme-apply may in turn call eval-args to evaluate the procedure call's arguments. FUNCTION eval-args node procedure args env This generic function is called by scheme-apply with an ordered-list of program-elements representing unevaluated arguments to a procedure. This generic function evaluates these arguments and returns an ordered list of instances scheme-value's, representing the evaluated procedure arguments.

eval-arg formal arg env This generic function is called by eval-args to evaluate an argument in a given environment. It returns a scheme-value representing the evaluation of the argument expression. 10

get-value binding This generic function is called to extract the value of a binding. A binding represents the association between a variable and a value. This method allows access to the point where the interpreter tries to retrieve the value of a binding.

set-value binding value This generic function is called whenever the value of a binding is to be updated. This generic function allows access to the point where the value of a binding is set.

make-binding variable value This generic function is called whenever a new binding is needed for the environment. It associates a variable with its value and returns this association in the form of an instance of binding. This generic function further allows for the programmer to specialize the type of binding created based on both the variable class and on the value class. This allows for aliasing and programmer control over interpreter state based solely on marks made on the program graph.

return-value value This generic function is called by scheme-eval with the value it is going to return. Allows for tracing of values being passed from one level to the next via scheme-eval.

pre-primop value This generic function is called whenever the canonical representation of a scheme-value is needed. Such a canonical representation is needed when arguments are applied to a strict operator, when the conditional of an if expression is being evaluated, or when an operator is applied. pre-primop is called on scheme-values in all such instances.

extend-env! binding env This function is called whenever a binding is to be added to an environment. For example, when a de ne expression is evaluated, a binding is added to the environment. The modi ed environment is then returned.

extend-env-for-proc bindings env This function is called whenever the environment needs to be extended for a procedure call, the evaluation of the body of a let-form, etc. This function returns a new environment containing the members of the list of bindings.

Appendix C { Source Code For Extensions In this section, we present a detailed description and some motivation for all of the semantic extensions which we have implemented. The source code for each implementation is also included. 11

Arbitrary Order of Argument Evaluation

The idea here is to control the order of evaluation of arguments to a call. There are two primitive mechanisms for providing this control. (i) A new class of call node, that provides control over order of evaluation in just that call. (ii) A new class of lambda node, which arranges for all calls on the closure to use a speci ed order. The rst requires only a very simple protocol. A specialization on scheme-eval is written which evaluates the arguments in a given order (reverse, or right to left in the simple case). This is implemented in example 1 below. The second requires that closures (the results of evaluating lambda nodes) have control over how the call happens and how the arguments are evaluated. Example 2 below demonstrates how the protocol must be re-implemented for modi cation of order of evaluation for speci c calls. ;;; Example 1: Make a call node evaluate its arguments in an arbitrary order. (defclass call-node-with-order-control (call-node) ((order :initarg :order :reader eval-order))) ;;; This specialization evaluates the arguments in the order ;;; specified by a user supplied list. (defmethod eval-args ((node call-node-with-order-control) (closure closure) args env) (let ((formals (closure-formals closure)) (copy (copy-list args))) (dolist (n (eval-order node)) (setf (nth n copy) (eval-arg (nth n formals) (nth n copy) env))) copy)) (defmethod eval-args ((node call-node-with-order-control) (primop primop) args env) (let ((copy (copy-list args))) (dolist (n (eval-order node)) (setf (nth n copy) (scheme-eval (nth n copy) env))) copy)) ;;; A simplified version of call-node-with-order-control which automatically ;;; evaluates the arguments from right to left. (defclass reverse-call-node (call-node) ()) (defmethod eval-args ((node reverse-call-node) (closure closure) args env) (let ((formals (closure-formals closure))) (reverse (mapcar #'(lambda (formal arg) (eval-arg formal arg env)) (reverse formals) (reverse args))))) (defmethod eval-args ((node reverse-call-node) (primop primop) args env) (reverse (mapcar #'(lambda (a) (scheme-eval a env)) (reverse args)))) ;;; Example 2: Lambda nodes which always evaluate their arguments (upon calling

12

;;; the closure) in reverse order. (defclass reverse-lambda-node (lambda-node) ()) (defclass reverse-closure (closure) ()) (defmethod scheme-eval ((node reverse-lambda-node) env) (let ((result (call-next-method))) (change-class result 'reverse-closure) result)) (defmethod eval-args ((node call-node) (closure reverse-closure) args env) (let ((formals (closure-formals closure))) (reverse (mapcar #'(lambda (formal arg) (eval-arg formal arg env)) (reverse formals) (reverse args)))))

Following is an example of how the above metacode might be used in a scheme program: (define (foo x y) (* x y)) (define (reverse-test1) {reverse-call-node}(foo (display 3) (display 4))) (define (reverse-test2) {call-node-with-order-control :order '(1 0)} (foo (display 3) (display 4))) (define bar {reverse-lambda-node}(lambda (x y) (* x y))) (define (reverse-lambda-test) (bar (display 3) (display 4)))

Active Variables

Design of a protocol to allow the introduction of active variables into scheme. This will allow the programmer to associate functions with a variable which may be run in any of the following situations: (i) When the variable is rst bound to its value (ii) Anytime the variable is read (accessed) (iii) Anytime the variable is written (through a de ne, letrec, or a set!). The metaprogrammer needs access to the representation of the environments of evaluation (based on reading or writing of environment values reader/writer functions would be invoked). The environment is represented by a list of binding objects. Each binding contains slots for the variable-binding-node (from the program graph) and the object containing its value. A problem we ran into in developing get-value and set-value was that their predecessors (envlookup-binding and env-set!) were passed actual variable names to perform their calculations. These new routines require actual objects. However, the type of object which might be passed to these routines was not homogenous. In the case of evaluation of letrec's and de ne's actual variablebinding-node's would be passed to lookup-binding since variable-access-node's are not available in these routines and yet they still need to extend the environment. As a solution to this problem, we created dummy instances of a variable-access-node and passed it to lookup binding with the correct name \ lled in" the name slot. This is a (perhaps necessary) hack by any de nition. Another problem here is that the function which is de ned to be called upon binding, reading, or writing of an active-variable must be written in common-lisp. While this may present some advantages, the uniformity of the programming language is violated in a sense. Of course, this can be avoided by the programmer of the protocol since scheme-eval can be called on the function 13

invocation rather than calling the common-lisp function, funcall. This is a symptom of the larger issue of having the interpreter, and thus the extensions in the meta-object protocol, implemented in a dierent programming language than the one being interpreted (CLOS versus Scheme). Given user access to these new metaobjects and the protocol (generic functions) which operate on these objects, it is then relatively easy to introduce active variables into our scheme interpreter. We introduce before-methods on get-value and set-value such that anytime these methods are invoked on an active-binding (which must have an active-variable associated with it) the appropriate function speci ed in the class de nition is called. Similarly an after-method on initialize-instance is de ned such that whenever an active-binding is created the appropriate bind-time function is called. Following is the metacode for the implementation of active variables: (defclass active-variable-binding-node (variable-binding-node) ((bind-function :initarg :bind-function :initform #'ignore-it :reader bind-function) (read-function :initarg :read-function :initform #'ignore-it :reader read-function) (write-function :initarg :write-function :initform #'ignore-it :reader write-function))) (defclass active-binding (binding) ()) (defmethod make-binding ((variable active-variable-binding-node) value) (make-instance 'active-binding :variable variable :value value)) (defmethod initialize-instance :after ((b active-binding) &key value) (funcall (bind-function (binding-variable b)) value)) (defmethod get-value :before ((b active-binding)) (funcall (read-function (binding-variable b)) b)) (defmethod set-value :before ((b active-binding) value) (funcall (write-function (binding-variable b)) b value)) (defun ignore-it (&rest ignore) ignore)

Sample use of active-variables: (let (({active-variable-binding-node :read-function #'(lambda (b) (print b)) :write-function #'(lambda (b nv) (format t "Changing X to ~S." (scheme-to-common-lisp nv)))} x 1)) x

14

(set! x 2) x)

Partial Closures

The idea here is to allow the user to mark begin's as either de nitions of a partial closure or a partial (to full) completion of one of these partial closures. This is accomplished by having the user specify which variables are to be left open in the environment. The environment is then extended by *open-bindings* (a subclass of binding). If the programmer attempts to access the values of one of these open-binding's an error is indicated. When the programmer closes a partial-closure, a closure is returned equivalent to the original but with the open-bindings indicated by the variable list of the closer command replaced by normal binding's which can then be accessed in the standard fashion. Once again, no new protocol is introduced here. The same tricks used in getting active-variables to work (a specialization on the binding class, a new type of variable-binding-node (an openvariable-binding-node which signals the creation of an open-binding), and a new type of begin-node which is guaranteed to evaluate to a closure) are used to get partial closures to work here. Following is the metacode implementing partial closures: (defclass opener (begin-node) ((vars :initform () :initarg :vars :reader open-vars))) (defclass closer (begin-node) ((closes :initform () :initarg :vars :reader close-vars))) (defclass open-variable-binding-node (variable-binding-node) ()) (defclass open-binding (binding) ()) (defmethod scheme-eval ((open opener) env) (let ((bindings (make-bindings (mapcar #'(lambda (s) (make-instance 'open-variable-binding-node :name s)) (open-vars open)) (make-list (length (open-vars open)))))) (call-next-method open (extend-env-for-proc bindings env)))) (defmethod make-binding ((variable open-variable-binding-node) value) (make-instance 'open-binding :variable variable :value value)) (defmethod get-value ((b open-binding)) (barf)) (defmethod set-value ((b open-binding) new) (barf))

15

(defmethod scheme-eval ((closer closer) env) (let ((closure (call-next-method))) (make-closure (closure-formals closure) (closure-body closure) (let ((env-copy (copy-list (closure-env closure))) (new-env nil)) (dolist (close (close-vars closer)) (dolist (frame env-copy) (let ((open (find close frame :key #'(lambda (b) (when (typep b 'open-binding) (variable-binding-node-name (binding-variable b))))))) (when open (setq frame (subst (lookup close env) open frame)))) (setq new-env (cons frame new-env)))) (reverse new-env))))) (defun lookup (name env) (let ((var (make-instance 'variable-access-node :name name))) (lookup-binding var env)))

Sample Scheme code using partial closures: (define pc nil) (define setter nil) (define c nil) (let () (set! pc {opener :vars '(x)}(begin (lambda () x)))) (let ((x 1)) (set! setter (lambda (new) (set! x new))) (set! c {closer :vars '(x)}(begin pc)))

Memoization

Generic functions and metaobjects to allow for memoization. The general idea is that certain procedure de nitions are marked as memoize-de ne's (a subclass of the de ne-node). The closure bound to the de nition variable is a \memoize-closure". When scheme-apply gets called with a memoize-closure, it evaluates its arguments, and then checks to see if it has evaluated the procedure with those same arguments before. If so, it returns the memoized value (stored in a hash table). Otherwise, the body of the de nition is evaluated and the result is then stored in the hash table (for future reference). 16

NOTE: procedures which are called to change some state or whose return value is dependent on some external state should not be memoized. Following is the metacode necessary for the implementation of memoization: (defclass memoize-closure (closure) ()) (defclass memoize-define (define-node) ()) (defvar *closure-to-value* (make-hash-table :test #'equal :size 1000)) (defmethod scheme-eval ((node memoize-define) env) (let ((var (conn-other-node (variable-binding-conns node))) (result (call-next-method))) (change-class (get-value (lookup-binding var env)) 'memoize-closure) result))

;;; the body of this when statement basically reimplements the default ;;; implementation of scheme-apply. call-next-method could have been ;;; used to do this, but then the actuals to the procedure would have to ;;; be recomputed, and this might be a potentially expensive operation. (defmethod scheme-apply ((node call-node) (closure memoize-closure) args env) (let* ((actuals (eval-args node closure args env)) (key (cons closure actuals)) (result (gethash key *closure-to-value*))) (when (not result) (let* ((env (closure-env closure)) (body (closure-body closure)) (bindings (make-bindings (closure-formals closure) actuals)) (new-env (extend-env-for-proc bindings env))) (setq result (scheme-eval body new-env)) (setf (gethash key *closure-to-value*) result))) result))

Here is an example of the use of memoization: {memoize-define}(define (factorial n) (if (= n 1) 1 (* n (factorial (- n 1)))))

Normal Order Evaluation

The idea here is that the formals of a closure need to be consulted before the arguments to a call are actually evaluated. If the formal is a \delayed" node, then create a thunk (which is only forced 17

when it needs to be: upon contact with a primitive operation), otherwise evaluate the argument normally. eval-arg is called from scheme-apply with a formal, an unevaluated argument, and an environment. It is the responsibility of this function to return the evaluated argument; it is specialized on the formal. This new function made the implementation of normal-order evaluation fairly easy as we only needed a specialization on this function (eval-arg) to delayed-variable-binding-node's to create the appropriate thunks (thus eliminating the need to have the user mark lambda's and the need to introduce delayed-closure's). Following is the code for normal-order evaluation: (defclass delayed-var (variable-binding-node) ()) (defclass thunk (scheme-value) ((expr :initarg :expr :accessor thunk-expr) (env :initarg :env :accessor thunk-env))) (defun make-thunk (expr env) (make-instance 'thunk :expr expr :env env)) (defmethod eval-arg ((formal delayed-var) arg env) (if (typep arg 'thunk) arg (make-thunk arg env))) (defmethod pre-primop ((val thunk) env) (scheme-eval (thunk-expr val) (thunk-env val))) (defmethod scheme-eval ((node thunk) env) node) (defmethod do-print ((form thunk)) (do-print (scheme-eval (thunk-expr form) (thunk-env form))))

Here is an example usage of normal-order evaluation from SICP: (define (unless pred {delayed-var}default-action {delayed-var}exception) (if (not pred) default-action exception)) (define (factorial n) (unless (= n 1) (* (factorial (- n 1)) n) 1)) (define (try {delayed-var}a {delayed-var}b) (if (= a 0) 1 b)) (try 0 (/ 1 0))

18

Fluid Let

To implement uid let's, another method on scheme-eval is introduced for a uid-let-node and two auxilary functions are used to clarify the implementation of this new method. The idea is to allow the user to mark let-node's as being special. All bindings within uid-let's are assumed to be special in the following way. The variable must have been bound in a frame above the uid let. If it was not, then an error is signalled, otherwise, the old value of the binding is saved and the binding is reset to re ect the new value speci ed by the uid let. The body of the uid-let is then evaluated in this new environment (where certain values where replaced). The key is that these values are replaced for all references to the new binding variables within the dynamic scope of the uid-let. Upon completion of the uid-let's body the saved values of the bindings are restored. Following is the metacode for the implementation of uid-lets: (defclass fluid-let-node (let-node) ()) (defun store-and-update-env-vals (formals new-vals env) (mapcar #'(lambda (formal new-val) (let* ((formal-binding (lookup-binding formal env)) (old-val (get-value formal-binding))) (set-value formal-binding new-val) old-val)) formals new-vals)) (defun restore-values (formals vals env) (mapc #'(lambda (formal val) (set-value (lookup-binding formal env) val)) formals vals)) (defmethod scheme-eval ((node fluid-let-node) env) (let* ((formals (mapcar #'conn-other-node (variable-binding-conns node))) (values (mapcar #'conn-other-node (init-conns node))) (body (conn-other-node (body-conn node))) (actuals (mapcar #'(lambda (value) (scheme-eval value env)) values)) (former-vals (store-and-update-env-vals formals actuals env)) (return-val (scheme-eval body env))) (restore-values formals former-vals env) return-val))

Example usage of uid let's: (define x 7) (define bad (lambda () x)) (define foo (lambda () (let ((x 5)) (letrec ((bar

19

(lambda () {fluid-let-node}(let ((x 10)) (baz)))) (baz (lambda () x))) (display (bar)) (display (bad)) (baz)))))

Active Values

The introduction of active values lead to a number of fairly important changes to rst the protocol, second the way that objects are marked, and nally the way primitive values are absorbed from common-lisp. It became apparent that such changes were necessary after witnessing that three classes had to be introduced for active-pair values: active-binding's, active-call-node's, and the active-pair itself.

 Initially the creation of an \active-value" lead to the creation of an \active-binding". Anytime

the method get-value is called on an active-binding the function associated with the activebinding is invoked, calling the function associated with the active-binding. This is clumsy because in theory an active value should not be related to an active binding and also because the actual read-function associated with an active-value would have to be copied over to the active-binding. To circumvent this problem, a new layer, inspect-value, was introduced to the protocol. Inspect-value is called by scheme-eval with the value it is about to return. In the default case, the value is just returned without any side eects. A specialization on inspect value for active-values, however, allows the necessary function to be called by inspectvalue before returning the value itself. The introduction of this new method allowed for the elimination of the active-binding class.  When makring something of the form (cons 1 2) to be active, one is in fact marking a call-node when what one really wants is to mark the value returned by the call to be active. Previously another class, active-pair-call-node would have been introduced which would have called its next method and changed the class of its result to an active-pair. To solve this problem, all nodes were given an extra slot: rt-class (runtime class). Thus a mark of the form: {call-node :rt-class '(active-pair (read-function . value) (initarg2 . value2)} (cons 1 2)

now tells the interpreter (through an around method on scheme-eval, another addition to the interpreter) to change the class of the *result* of any call to scheme-eval whose node has a non-nil value for its rt-class slot (thus the object returned in evaluating the above cons would be an active-pair). Furthermore, a list of initargs are provided which are then added to this new class (in the above example the slot read-function would be set to value, the slot with the name initarg2 would be set to value2). This new form-of marking allows the programmer of the protocol to mark actual runtime objects rather than just program graph objects (thus going beyond just if-node's, let-node's, etc.) 20

 Finally, to allow the programmer to do specialized things with accessor's of values (such as

car and cdr which access pairs), generic functions were introduced for the primitive operators such as car, cdr, +, ?, >, etc. The user can then specialize on these methods to (for example) invoke the read-function of an active-value whenever the car of a pair is requested. The operands for these primitives is absorbed from common lisp. Thus, cl-value has been removed from the class of runtime objects and replaced by scheme-value, which is then further subclassed to num's, str's, bool's, and pairs. An equivalence relationship is set up between each scheme-value and its corresponding common lisp value through hash tables.

Following is the metacode for the implementation of active values: (defclass active-value (scheme-value) ((read-function :initarg :read-function :initform #'ignore-it :accessor read-function))) (defclass active-num (active-value) ()) (defmethod return-value ((node node) (val active-value)) (funcall (read-function val) val) (call-next-method)) (defun ignore-it (&rest ignore) ignore) (defclass active-pair (active-value) ()) (defclass change-class-begin-node (begin-node) ((rt-class :initform nil :initarg :rt-class :reader rt-class))) (defmethod scheme-eval ((node change-class-begin-node) env) (let ((result (call-next-method)) (new-class-info (rt-class node))) (let ((class-name (car new-class-info))) (change-class result class-name) ;; the cdr of new-class-info is a sequence of pairs containing ;; initialization information for the new-class. Loop through ;; the list, initializing the class as necessary. (dolist (currinit (cdr new-class-info)) (eval `(setf (,(car currinit) ,result) ,(cdr currinit))))) result))

Following is an example of active numbers: (defun test (x)

21

(print "hello world")) (define x {change-class-begin-node :rt-class '(active-num (read-function . #'test))} (begin 5))

Dependent Variables

Dependent variables are implemented as follows: A de ne-node is marked as de ning a dependent variable and given a list of variables upon which it is dependent. The binding of all variables which it is dependent on is then changed to a new class. A global variable is maintained which contains a list of all dependent variables. A specialization on set-value (which takes a binding and a value, and is specialized to dep-binding) checks the global list of variables for any matches between the current binding-variable and any of the variables within the various dependency lists of the global variable. If a match is found, then the entire body of the dependent variable is re-evaluated in its original environment. Following is the metacode for this implementation of dependent variables: (defvar *the-dependent-variables* (list '())) (defclass dep-define-node (define-node) ((depend-vars :initarg :depend-vars :accessor depend-vars))) (defclass dep-binding (binding) ()) (defmethod variable-node-name ((var-name symbol)) var-name) (defmethod scheme-eval ((node dep-define-node) env) (let* ((var (conn-other-node (variable-binding-conns node))) (body (conn-other-node (body-conn node))) (var-refs-binding (mapcar #'(lambda (var) (lookup-binding var env)) (depend-vars node))) (var-binding-nodes (mapcar #'binding-variable var-refs-binding))) (setq *the-dependent-variables* (cons (list var var-binding-nodes body env) *the-dependent-variables*)) ;; the change-class should eventually be changed to add-class (mapcar #'(lambda (binding) (change-class binding 'dep-binding)) var-refs-binding) (extend-env! (make-binding var *undefined-value*) env) (set-value (lookup-binding var env) (scheme-eval body env)) var)) (defun dep-variable-binding (dep-var) (car dep-var)) (defun dep-variable-references (dep-var) (cadr dep-var)) (defun dep-variable-body (dep-var)

22

(caddr dep-var)) (defun dep-variable-env (dep-var) (cadddr dep-var)) (defmethod set-value ((binding dep-binding) (new-value scheme-value)) (let ((var-to-set (binding-variable binding))) (setf (binding-value binding) new-value) (dolist (curr-var *the-dependent-variables*) (when (member var-to-set (dep-variable-references curr-var) :test #'eq) (set-value (lookup-binding (dep-variable-binding curr-var) (dep-variable-env curr-var)) (scheme-eval (dep-variable-body curr-var) (dep-variable-env curr-var))))) new-value))

Example scheme usage of dependent variables: (define y 4) (define z 3) {dep-define-node :depend-vars '(y z)}(define x (+ y z)) (set! y 8) (set! z (+ 4 5))

References [ASS85] Harold Abelson, Gerald Jay Sussman, and Julie Sussman. Structure and Interpretation of Computer Programs. MIT Press, Cambridge, Mass., 1985. [Cli91] The revised4 report on the algorithmic language Scheme, November 1991. [KdRB91] Gregor Kiczales, Jim des Rivieres, and Daniel G. Bobrow. The Art of the Metaobject Protocol. MIT Press, 1991. [KL92] Gregor Kiczales and John Lamping. Issues in the design and documentation of class libraries. In Proceedings of the Conference on Object-Oriented Programming: Systems, Languages, and Applications, 1992. To Appear. [Rod91] Luis H. Rodriguez Jr. Coarse-grained parallelism using metaobject protocols. Master's thesis, Massachusetts Institute of Technology, 1991.

23