Formalizing Neural Networks using Graph Transformations Michael R. Berthold
IRF (Prof. D. Schmid) University of Karlsruhe P.O. Box 6980, 76128 Karlsruhe, Germany
[email protected]
Abstract In this paper a unifying framework for the formalization of dierent types of Neural Networks and the corresponding algorithms for computation and training is presented. The used Graph Transformation System oers a formalism to verify properties of the networks and their algorithms. In addition the presented methodology can be used as a tool to visualize and design dierent types of networks along with all required algorithms. An algorithm that adapts network parameters using standard gradient descent as well as parts of a constructive, topology{changing algorithm for Probabilistic Neural Networks are used to demonstrate the proposed formalism.
1. Introduction
In the past decade many new types of Neural Networks along with new and variations of old algorithms for their training have been introduced. Formal ways to describe these networks and their training have been proposed by numerous researchers, but most of them are limited to a speci c type of network or algorithm. There still remains a need for a unifying framework which allows to formalize the network's structure together with all required algorithms, i.e. the computation of the network's activations and the training algorithm. This framework should oer ways to formalize proofs about the behaviour of the network as well as a way to easily create new types of Neural Nets together with their corresponding algorithms. In the past, dierent types of grammars have been used to generate families of Neural Networks; in [3] this approach together with a genetic algorithm was used to automatically nd the topology of a Neural Network. Freund and Haberstroh use Graph Grammars to formalize training algorithms for Multi Layer Perceptrons [4]. Lucas presents a formalism using Graph Grammars to grow adaptive Neural Networks [5], describing another way to construct a network automatically. Another interesting aspect is presented in [9], here a single{ ow graph representation is used to show
Ingrid Fischer
IMMDII University of Erlangen{Nurnberg Martenstr. 3, 91058 Erlangen, Germany
[email protected]
the equivalence of dierent learning algorithms. All of these approaches are limited, however, to topology{ preserving algorithms, i.e. the architecture of the network has to be determined up front. In this paper a uni ed formalism for the representation of the structure of Neural Networks is proposed. The presented framework can easily be used to formalize algorithms required to use and train the Network. The used Graph Rewriting Systems additionally provide a visual way to construct the networks and therefore enable rapid prototyping. Furthermore ways to formalize proofs about the network's behaviour and properties of the training algorithm (e.g. termination, convergence, etc.) are provided. In the following we explain the proposed formalism and demonstrate its usefulness on two well{known types of networks and two dierent types of algorithms, one that simply changes parameters of the network and another one that also adapts the network's topology during training.
2. Neural Networks seen as Graphs
Before formalizing the computation or training algorithms using Graph Rewriting Systems, the structure of the Neural Net has to be represented through a well{ de ned graph. Here a graph is seen as a quadrupel (E; V; s; t) with E (edges) and V (vertices or nodes) being sets. s : E ! V , t : E ! V denoting source and target are functions giving start and end node of an edge. Additionally edges and nodes can be labeled with elements of some labeling alphabets LE and LV using functions lE : E ! LE and lV : V ! LV . This can be summarized as follows: s l LE lE E - V V- LV t For the labeling of edges we will use pairs of real numbers of < and functions (F = ff : < ?!