s -lV

3 downloads 0 Views 210KB Size Report
Formalizing Neural Networks using Graph Transformations. Michael R. Berthold .... e(?x2=2),. 1=(1 + e? x),. lV (vI) = eucl. distance weighted sum. lV (vA) = Gaussian act. ... well known sigmoidal squashing function would be used for the MLP ...
Formalizing Neural Networks using Graph Transformations Michael R. Berthold

IRF (Prof. D. Schmid) University of Karlsruhe P.O. Box 6980, 76128 Karlsruhe, Germany [email protected]

Abstract In this paper a unifying framework for the formalization of di erent types of Neural Networks and the corresponding algorithms for computation and training is presented. The used Graph Transformation System o ers a formalism to verify properties of the networks and their algorithms. In addition the presented methodology can be used as a tool to visualize and design di erent types of networks along with all required algorithms. An algorithm that adapts network parameters using standard gradient descent as well as parts of a constructive, topology{changing algorithm for Probabilistic Neural Networks are used to demonstrate the proposed formalism.

1. Introduction

In the past decade many new types of Neural Networks along with new and variations of old algorithms for their training have been introduced. Formal ways to describe these networks and their training have been proposed by numerous researchers, but most of them are limited to a speci c type of network or algorithm. There still remains a need for a unifying framework which allows to formalize the network's structure together with all required algorithms, i.e. the computation of the network's activations and the training algorithm. This framework should o er ways to formalize proofs about the behaviour of the network as well as a way to easily create new types of Neural Nets together with their corresponding algorithms. In the past, di erent types of grammars have been used to generate families of Neural Networks; in [3] this approach together with a genetic algorithm was used to automatically nd the topology of a Neural Network. Freund and Haberstroh use Graph Grammars to formalize training algorithms for Multi Layer Perceptrons [4]. Lucas presents a formalism using Graph Grammars to grow adaptive Neural Networks [5], describing another way to construct a network automatically. Another interesting aspect is presented in [9], here a single{ ow graph representation is used to show

Ingrid Fischer

IMMDII University of Erlangen{Nurnberg Martenstr. 3, 91058 Erlangen, Germany

[email protected]

the equivalence of di erent learning algorithms. All of these approaches are limited, however, to topology{ preserving algorithms, i.e. the architecture of the network has to be determined up front. In this paper a uni ed formalism for the representation of the structure of Neural Networks is proposed. The presented framework can easily be used to formalize algorithms required to use and train the Network. The used Graph Rewriting Systems additionally provide a visual way to construct the networks and therefore enable rapid prototyping. Furthermore ways to formalize proofs about the network's behaviour and properties of the training algorithm (e.g. termination, convergence, etc.) are provided. In the following we explain the proposed formalism and demonstrate its usefulness on two well{known types of networks and two di erent types of algorithms, one that simply changes parameters of the network and another one that also adapts the network's topology during training.

2. Neural Networks seen as Graphs

Before formalizing the computation or training algorithms using Graph Rewriting Systems, the structure of the Neural Net has to be represented through a well{ de ned graph. Here a graph is seen as a quadrupel (E; V; s; t) with E (edges) and V (vertices or nodes) being sets. s : E ! V , t : E ! V denoting source and target are functions giving start and end node of an edge. Additionally edges and nodes can be labeled with elements of some labeling alphabets LE and LV using functions lE : E ! LE and lV : V ! LV . This can be summarized as follows: s l LE lE E - V V- LV t For the labeling of edges we will use pairs of real numbers of < and functions (F = ff : < ?!