Network access protocols for real-time distributed

IEEE TRANSACTIONS ON INDUSTRY APPLICATIONS, VOL. 24, NO. 5, SEPTEMBERIOCTOBER 1988

897

Network Access Protocols for Real-Time Distributed Control Systems ASOK RAY,

SENIOR MEMBER, IEEE

Abstract-Analysis and design for real-time distributed data communication and control system networks is an evolving field which requires interactions between the disciplines of communication systems and control systems engineering. Concepts of pertinent network parameters are introduced and an analytical base for system design is establisbed. A simulation methodology has been developed for performance evaluation of real-time control system networks for distributed dynamical processes like autonomous manufacturing, electric power and chemical plants, and aircraft and spacecraft. The network-induced delays, resulting from data latencies, data loss, and missynchronism between system components, degrade the dynamic performance of the controlled process and are a source of potential instability. The impact of delays induced by different network access protocols on the control system performance are investigated. The results of combined discrete-event and continuous-time simulation are presented to this effect.

I. INTRODUCTION ISTRIBUTED information processing is essential for control and operation of complex dynamical processes like autonomous manufacturing, large-scale electric power and chemical plants, and advanced aircraft and spacecraft. This requires a number of spatially dispersed computers and intelligent terminals to perform, in real time, diverse but interrelated functions ranging from closed-loop control of essential process variables to routine maintenance support and information display [ 11-[4]. The requirements for a communication network for a distributed control system may vary for specific applications. For example, in an autonomous manufacturing environment, the prime objective of distributed data communication and control system (DDCCS) networks is the integration of computer-controlled complexes of robots, automated machine tools, material-handling devices, and guided vehicles on the shop floor, with information processing across local and remote stations. Similarly, in nuclear power plants, distributed control systems could regulate the essential and time-dependent process variables, like reactor thermal power, feedwater flow, and electric power, as well as provide the necessary support for plant monitoring, fuel scheduling, and preventive maintenance. The major challenge in an advanced

D

Paper IUSD, approved by the Industrial Control Committee of the IEEE Industry Applications Society for presentation at the 1986 Industry Applications Society Annual Meeting, Denver, CO, September 28-October 2. This work supported in part by the National Science Foundation under research grant DMC-8707648. Manuscript released for publication December 1, 1987. The author is with the Mechanical Engineering Department, 208 Mechanical Engineering Building, The Pennsylvania State University, University Park, PA 16802. IEEE Log Number 8820182.

aircraft is to design a fly-by-wire (or fly-by-light) integrated control system that will ensure safe and reliable missions. Two basic methods are available for interconnecting the components of a distributed information-process and control system [3], [4]: 1) point-to-point dedicated connections, and 2) asynchronous time-division multiplexed (TDM) networking. The major advantages of multiplexed networks over pointto-point connections include reduced wiring and power requirements, flexibility of operations, evolutionary design process, and ease of maintenance, diagnostics, and monitoring. However, the data latency in multiplexed communication networks could introduce unacceptable delays and thereby degrade the dynamic performance of the control system. The sensor and control data in a DDCCS network are subject t 3 time-varying delays due to data latency of messages in the communication network [5], [6] in addition to the sampling time delay that is inherent in digital control systems. This is evident in very fast processes (e.g., flight dynamics in tactical aircraft [5]). The effects of data latency on the dynamic performance and stability of feedback control systems are often ignored in relatively slow processes, such as those encountered in manufacturing and processing plants. However, as the number of users on the network increases, the augmented traffic causes a larger data latency, to a point where its impact on the performance of some of the control loops (sharing the network) can no longer be ignored. The detrimental effects of data latency on the system dynamic performance are further aggravated by missynchronisni between the control-loop components as well as by loss of messages resulting from the saturation of buffers at the terminals and the data corruption by noise in the network medium. In a distributed control system environment, the network traffic would consist of both real-time and non-real-time data packets. The real-time data are usually generated as short messages and must be communicated on a periodic basis as the sampling interval T of a control system is predetermined according to the dynamic characteristics of the controlled process. For example, T could be very small, that is, on the order of 50 ms, for communications between two robots working on the same transfer station. The non-real-time traffic is generated by the design and administrative information, such as production orders, part drawings, and status reports, and is transmitted nonperiodically and much less frequently. These messages are usually long and do not need to be processed within the constraints of a real-time environment. Whereas data latency is critical for real-time messages, non-real-time mes-

0093-9994/88/0900-0897$01.00

O

1988 IEEE

898

-

IEEE TRANSACTIONS ON INDUSTRY APPLICATIONS, VOL. 24, NO. 5 , SEPTEMBERIOCTOBER 1988

sages need the assurance of accurate delivery via the shared communication medium. Occasional losses of (real-time) sensor and control data in a feedback loop can be tolerated whereas the additional delay due to retransmission of erroneous signals may seriously degrade the system dynamic performance. Therefore the realtime data may not use the acknowledgment option; messages with detected errors would be simply discarded. On the other hand, the acknowledgment option needs to be adopted for non-real-time messages to ensure accurate delivery of data. Under these circumstances the real-time data should have a higher priority over the non-real-time data. This may delay the delivery of the non-real-time data but will not comprise the data integrity because of the built-in packet retransmission capabilities of the network protocol. The mixing of real-time unacknowledged data and delayable acknowledged data is similar to what is encountered in packetized voice and data communication networks [7]. The preceding concept can be extended to integrated voice and data communications if the real-time data packets are replaced by voice packets. Analysis and design of an integrated network for real-time DDCCS require interactions between the disciplines of communication systems and control systems engineering. It may be appropriate to bring out the notions of delay, as it is used in the two disciplines in somewhat different manners. In communication systems, the delay is primarily referred to as queueing delay and data latency, which are associated with only those messages that successfully arrive at the destination terminal [5]; messages that are corrupted by noise or deleted due to queue saturation at the transmitter buffer of the source terminal are not considered for this purpose. In real-time control systems the delay is related to the question: How old is the data that is currently used?. When no messages are rejected, the two notions of delay are identical; otherwise they are different. Although sufficient research papers in modeling and simulation of communication protocols have been published [8], the significance of network-induced delays relative to the stability of feedback control systems apparently has not been addressed except in a few cases [5], [6]. A systematic methodology needs to be developed for design of real-time control systems that are subject to network-induced delays. A generic methodology, based on simulation analysis, has been developed for evaluating the system dynamic performance. Further analytical work in this evolving area is under active research [9], [lo]. The major objectives of this paper are to analyze the characteristics of network-induced delays in distributed communication and control systems; and to introduce a simulation methodology for evaluating the system dynamic performance and present pertinent results of simulation experiments for diverse network traffic. The paper is organized in five sections. Section 11 introduces concepts of pertinent network parameters and establishes an analytical base for system design. The structure of a simulation methodology that has been developed for performance evaluation of real-time distributed control systems

is described in Section 111. Simulation resiilts under diverse traffic are presented and discussed in Section IV. The summary and conclusions of this evolving research are presented in Section V. OF DISTRIBUTED 11. PERFORMANCE ANALYSIS CONTROL SYSTEM NETWORKS

This section emphasizes analytical modeling of networkinduced delays with the objective of evaluating the dynamic performance of distributed control systems. Other pertinent parameters for DDCCS performance evaluation, such as network reliability and data frame error rate, have not been addressed in this paper. The reason is that reliability and availability are largely hardware-dependent and are usually analyzed during the hardware design phase, when detailed specifications become available. Data frame errors depend on the raw bit error rate in the network medium and the error detection algorithm provided in the protocol (e.g., cyclic redundancy check). Raw bit error rates in (coaxial cable) network media in a typical industrial environment range from lo-” to [ l l ] ; corresponding values for optical fiber media are significantly lower. Although message retransmissions or rejections (as a result of detected frame errors) apparently have no significant bearing on data latency and throughput, under these circumstances it is the undetected frame errors that degrade control system reliability. Therefore the issue of undetected frame error rate should be examined along with reliability. The following conditions were assumed for analytical modeling of the DDCSS network. Each terminal has one transmitter queue and one receiver queue. First in, first out (FIFO) ranking is adopted at each terminal’s transmitter queue, and its buffer capacity is limited to an a priori assigned constant; in the event of queue saturation, the first message (i.e., the message at the front end of the queue) is rejected and the stack is pushed down to accommodate the new arrival at the rear end of the queue. The receiver queue capacity at each terminal is one. Now we proceed to develop analytical models for network performance evaluation by introducing pertinent definitions. Definition I : Network traffic is defined to be deterministic if the message length and the message interarrival time are constants. Definition 2: Network traffic is defined to be simultaneous if it is deterministic and if the messages arrive at all terminals simultaneously at each sampling period. Definition 3: For a message, the word count Wc is defined as int (L/Wd), if rem ( L ,

WC(L) =

wd)= 0

int ( L / W d )+ 1, if rem (L,

>0

wd)

where

L

length (in bits) of the data part of a message,

899

RAY: NETWORK ACCESS PROTOCOLS

wd

int(*)

excluding the bits due to formating and overhead, number of data bits per word, the integer part of *

and rem (a, b) P a - [int (a, B)]b. Definition 4: Frame length L‘ (bits) of a message is defined as

L‘

=

W f W,(L)

+Q

where

W f length (bits) of a formated word,

Q

overhead (bits) associated with a message.

Definition 5: Offered traffic G is defined as G = (E[L]N)/RE[T]) where N

number of active terminals in the network,

data transmission rate in bitsh, R message interarrival time at a terminal, T E[*] expected value of *.

Definition 6: Cycle time

7

is defined as

terminal and the instant of reception of its last bit at the destination terminal. Remark 4: Data latency and queueing delay are approximately related as

6 = 6,

+ L’/R

where R equals the data transmission rate in bits/s. In a feedback control loop, messages are transmitted via the network medium from the sensor terminal to the controller terminal and from the controller terminal to the actuator terminal. Thus the control system in Fig. 1 is subject to time-varying delays due to data latency, and its dynamic performance is dependent on the traffic in the network. Additional definitions of network delay parameters are introduced below. of a successDefinition ZZ: Sensor-controller delay ful message at the kth sensor sampling instant is defined as the interval between the instant of its arrival at the sensor terminal’s transmitter buffer and the instant when the controller starts processing this message. Remark 5: If the sensor and controller have the same sampling time T, e,, can be expressed in terms of the sensorto-controller data latency s,, and the time skew As between the sensor and controller sampling instants (0 < A, 5 T) as

n

7 =

C [ L { / R ]+ NO i= I

where u equals the average bus idle time prior to the beginning of a message transmission and subscript i corresponds to the terminal i. Remark I : For a given traffic, cycle time may be interpreted as the token time required to complete the transmission of one message from each of the N terminals. Definition 7: For a given traffic and normalized cycle time, G’ is defined as the ratio of the expected values of the cycle time and the message interarrival time, i.e., G‘ =

E[7]/E[T I . Remark 2: For a given G, individual protocols may load the medium to different levels and thus influence the performance of the DDCCS to different degrees. Therefore G is used as a parameter for selection of network access protocols instead of G’. A limit of G above which a given protocol is expected to overload the medium, resulting in message rejection, needs to be specified. Definition 8: The critical offered traffic G,, for a protocol is defined as the largest offered traffic for which no message frame is rejected, due to queue saturation for deterministic traffic under steady state. Remark 3:

G’ I1 if G = G,,

Definition 9: Queueing delay 6, of a successfully transmitted message is the difference between the instant of arrival of the message at the transmitter queue of the source terminal and the instant of transmission of its first bit on the medium. Definition 10: Data latency 6 of a successfully transmitted message is defined as the difference between the instant of arrival of the message at the transmitter queue of the source

e,C(6,,,

As)

=I

As for 6,, < As f f + A s for ( P - 1 ) T + A S ~ 6 , c P T + A s

where P is a positive integer. of a successDefinition 12: Controller-actuator delay ful message at the jth controller sampling instant is the interval between the instant when the controller starts processing the available sensor data and the instant of arrival of the last bit of the processed control command at the actuator. Remark 6: In contrast with the sensor data, which may wait at the controller’s receiver queue before being processed by the controller, the control command acts upon the plant immediately after arriving at the actuator terminal. This happens because the controller is scheduled to generate signals at constant intervals whereas the actuator operations are essentially asynchronous. Remark 7: Since the delay is time-varying, the controller may use the sensor data generated at the current or earlier samples. The delay e,, to which the control command sequence is subject is also time-varying. This implies that the intervals between successive arrivals of control commands at the actuator terminal may not be a constant. Remark 8: The characteristics of the delay Os, are different from those of e,, in the case where more than one sensor data arrive at the controller during one of its sampling intervals. Since the controller operates in discrete time, unlike the actuator, which is essentially a continuous-time device, only the most recent sensor data is accepted. This phenomenon is called message rejection at the controller’s receiver. On the other hand, if no fresh sensor data is available during a sampling interval, the previous sensor data is used

e,,

900


long as t that

t

Fig. 1 .

Delayed control system.

J

=k

Fig. 2 . Lumped delayed system.

< (k + l)T +

- X(t) =

kT

+

C#I

- h(kT

This follows from the fact

+ 4) 2 kT

at the instant t = kT + 9. Thus the input to the plant is u(t) = uk. Remark 10: As can be seen from the conditions 4) and 5) in Proposition 1 , h(t) is not unique, and any admissible function is equally applicable. Remark 11: If X(t) is a constant, or can be approximated by a constant for all t, even though 8, and 8, could be individually time-varying, then the control system can be designed using conventional frequency-domain techniques.

Proposition 2: For periodic traffic and G > G,,, Le., to compute the control command. This phenomenon is called r < T, implying that some messages are rejected due to vacant sampling at the controller. Remark 9: Since the delay is time-varying, the con- queue saturation, let r = mT + 6 , where m is a positive troller may use the sensor data generated at the current or integer and 6 E (0, TI. Then earlier samples. The delay e,, to which the control command a) the probability of message rejection is ( r - T ) / r ; sequence is subject is also time-varying. This implies that the b) following a successful message either (m - 1) or m intervals between successive arrivals of control commands at messages are rejected. The probability of m rejections the actuator terminal may not be a constant. isP, = 6 / T . Under random traffic conditions the control system delays Proof of Proposition 2a): The expected value of the e,, and e,, are stochastic processes. Even for deterministic traffic, i.e., periodic traffic with constant message lengths, number of messages that arrive at each terminal during the interval Tr is r. The expected value of the number of mesthese delays are both time-varying and therefore, in general, sages that are successfully transmitted during the same period may not be lumped together. However, if the digital control algorithm is linear time-invariant, these two delays could be is T . Hence the probability of a message to be successfully transmitted in T/r and that of rejection is ( r - T)/r. lumped under certain conditions. A statement of this property is given below. Proof of Proposition 26): Let the messages periodically Proposition 1: The time-varying delays e,, and e,, in arrive at a certain terminal at the instants j T . Upon successful Fig. 1 can be equivalently lumped together as X in Fig. 2, transmission of a message at tf = kT + a,a E (0, TI, the provided that next successful message is transmitted at I f + r = ( k + m)T + a + 8. If (a + 6) > T , then m messages are rejected; 1 ) the sensor and controller have identical sampling periotherwise, (m - 1) messages are rejected. ods T , 2 ) there is no message rejection at the sensor and con- To the obtain the probability of message rejections, we make troller terminals, use of the following lemma to show that the probabilities of 3) X(t) 2 0 for v t , the events {(a 8 ) > T} and {(a 6 )IT} are 29/Tand 4) h(kT ak)= ak,and ( T - 6 ) I T , respectively. 5 ) X(kT 9) I q5 V 4 E (ak, T a'+'), Lemma for Proposition 2: For r > T, if CY, is defined as

e,,

+

+

where

a, = rem (ao j r , T )

ak = e:, + e:,.

where rem

Proof of Proposition 1: The lumped and unlumped delayed systems in Figs. l and 2 are equivalent with respect to the input/output relation if, for a given input y ( t ) to the controller, the outputs u(t) and u*(t) are identical. The function u(t) is piecewise constant and is given by

u(t) =

uk

if k T

+ ak < t I( k + l ) T + @+'.

The function u * ( t ) is given by

u*(t) =

+

+

+ +

uk

if k T S (t - h(t))< ( k

+ l)T

For u(t) and u*(t) to be equivalent, the time ranges in the preceding equations must be the same. Condition 4 guarantees U(kT

ak)= uk

and condition 5 ) assures that this value remains constant as

( e ,

e )

is as stated in Definition 3, then

i) a0 = 0 implies that a, is discretely uniformly distributed in (0, T I ; ii) if a0 is continuously uniformly distributed in (0,T I , then aj is also continuously uniformly distributed in (0, TI.

Proof of Lemma i): Let y be the greatest common divisor of r and T , i.e., r = l y and T = 6 y , where 1 and 6 are co-prime integers and I > 6 .Then a, = rem ( j l , S)y and consequently a, assumes 6 distinct values j y for j = 1 , 2 , . * , 6. Since CY@+; = ai, the sequence {aj}is cyclic, with a period of 6,, and the distribution of ai is discretely uniform. Proof of Lemma ii): In this case a, assumes the 6 distinct values of a. + k, with equal probability, where - int [ao/yl 5 k 5 6 - int [ a O / y ]- 1. Since a. is continuously

90 1


uniformly distributed in (0, TI, uniformly distributed in (0, TI.

cyj

is also continuously

111. SIMULATION MODELS FOR DDCCS NETWORK PERFORMANCE EVALUATION Protocols based on asynchronous time-division multiplexing (TDM) [ l l ] are suitable for real-time DDCCS networks that are subject to a combination of periodic, nonperiodic, and bursty traffic. The advantages of distributed controlled-access protocols, like token bus and token ring, over random-access protocols, like CSMAKD, have been presented in our recent publications [ 5 ] , [6]. Two standard distributed controlled-access protocols, namely the Society of Automotive Engineers (SAE) token ring [12] and linear token bus [13], which offer larger data communication speeds and less complexity than the respective IEEE Std. 802 family protocols [14], [15], were selected as the candidate protocols for the DDCCS network. The performance of the SAE linear token bus and token ring protocols was compared with respect to the (centralized controlledaccess) MIL Std. 1553B protocol [16], which has been used extensively in digital avionic systems of fly-by-wire military aircraft control systems. Simulation modes for the SAE linear token bus and token ring and the MIL Std. 1553B protocols were developed for DDCCS performance evaluation using combined discreteevent and continuous-time techniques. Discrete-event models are suitable for representing the time-ordered sequences of operations that are encountered in network protocols. On the other hand, continuous-time models are essential for solving the differential and difference equations associated with plant and controller dynamics. The coupling of two types of simulation permits performance evaluation of the network as a dynamic element of the closed-loop control system. This allows for monitoring the effects of delays introduced by the network. The simulation program for the DDCCS consists of two subsystem models: 1) a discrete-event model of the network and 2) a continuous-time model of process dynamics and a discrete-time model of the controller. The program structure is modular, Le., any one of the protocol models (e.g., SAE linear token bus, SAE token ring, or MIL Std. 1553B) can be inserted in the simulation program while operating on the same control system model, and vice versa. A schematic diagram for the combined discrete-event and continuous-time model structure of the DDCCS network is shown in [6]. The network subsystem model consists of two independent but interacting submodels: 1) the message generation submodel and 2 ) the protocol submodel. The message-generation submodel has an identical structure for all types of protocols and is driven by an external pool of messages that arrive at the network system either periodically or at random intervals of time (Poisson arrival, for example). Similarly, the message lengths can be either constants or randomly distributed (exponential, for example). When a new message arrives at the system from the external message generator, the message attributes are defined to establish the message identity in the following ways.

Time of arrival- this is the instant at which the arrival of a message at the transmitter queue is recorded. The message information length (overhead not included). The source terminal, i.e., the terminal from which the message is generated. The destination terminal- this could be any terminal on the network other than the source terminal. The message priority, if applicable. The protocol submodel essentially represents the algorithm of the network access protocol under consideration. Different submodels, each of which identically enters the simulation program as a subroutine, have been developed for SAE linear token bus, SAE token ring, and MIL Std. 1553B protocols. Although internal algorithms of the individual protocol submodels are different, their interactions with the message generation submodel and the models of the plant and controller are identical. For example, the attributes of the generated messages are captured by the protocol submodel, which in turn regulates the delays for exchange of messages between the plant and controller terminals. The interactions between the plant and controller models involve exchange of sensor and control signals which undergo time-varying delays introduced by scheduled events in the network model. Key considerations in the choice of a language for the DDCCS simulation are: 1) combined discrete-event and continuous-time simulation capability; 2) programming flexibility and software portability; 3) verification and debugging capability; 4) built-in statistical testing capability; and 5) automatic ordering of scheduled events. After considering a number of simulation languages [17], SIMAN, which offers all of these key features [ 181, was chosen as the software for coding and implementation of the DDCCS model. IV. SIMULATION RESULTS AND DISCUSSION The network model was operated under steady-state conditions while disturbances were applied to the control system model to observe the impact of the network-induced delays on system dynamics. The following network configuration was employed as the basic model for simulation experiments and subsequent comparisons of the results. The network consists of 31 terminals or nodes (this is the upper limit for MIL Std. 1553B). Terminal 1 operates as the sensor and actuator terminal, with its transmitter queue serving the sensor and its receiver queue serving the actuator in the closed-loop control system. Terminal 2 operates as the controller terminal, with its transmitter queue handling actuator commands and the receiver queue handling sensor data. Terminals 1 and 2 have periodic traffic with fixed-length messages, with the data part L = 64 bits and a sample period of 10 ms. Terminals 3 to 31 are modeled as ordinary nodes of the network, where the expected value of the message interarrival time was set to 10 ms and message lengths were varied to regulate the offered traffic in the network. The digital control system is single-input single-output with unity feedback. The plant transfer function in Fig.

902

E E E TRANSACTIONS ON INDUSTRY APPLICATIONS, VOL. 24, NO. 5 , SEITEMBEWOCTOBER 1988

TABLE I STEADY-STATE PERFORMANCE OF NETWORK ACCESS PROTOCOLS

Average Data Latency ( p )

G Q Q

= 1 = 2

=

0.2

MIL Std. 1553B G = 0.7 G = 1.2

3231 3231

5308 15 309

5508 15 508

SAE Linear Token Bus G = 0.2 G = 0.7 G = 1.2 1077 1077

3659 3659

G

=

SAE Token Ring 0.2 G = 0.7 G = 1.2

5404 15404

1116 1116

3690 3690

5407 15 407

5338

1123 1123

3714 3738

5334 15 228

Exponentially Distributed Message Length and Constant Interarrival Time Q = 1 Q = 2

3101 3101

5258 15 240

5477 15 482

1079 1079

3687 3708

15 178

Constant Message Length and Exponentially Distributed Interarrival Time

Q

= 1

Q = 2

2095 3367

3842 9393

110 110

6206 15 555

499 567

2338 7684

127 127

508 577

2356 7720

135 136

676 835

2535 8096

Exponentially Distributed Message Length and Exponentially Distributed Interarrival Time

Q = 1 Q= 2

2066 3260

3807 9189

5914 15 131

+

118 119

+

1 was selected as Gp(s) = 1/[(0.3s 1)(0.03s l)], and the analog equivalence of the transfer function of the digital controller as Gc(s) = 7(s 5)/s, where s is the Laplace transform variable.

+

A . Steady-State Performance of the Network

668 820

2500 7899

is improved with larger variability in traffic. The randomness of T contributes more to performance improvement than that of L. However, no general conclusion can be drawn for G > G,,, especially if G significantly larger than Gcr. SAE linear token-bus and token-ring protocols yield almost equivalent performances, which are superior to that of the MIL Std. on the average. In terms of average data latency, the SAE linear token bus and token ring exhibit their best performance under Poisson arrival of messages with constant lengths. Results indicate that the linear token-bus and token-ring protocols clearly excel beyond the MIL Std. 1553B in terms of average data latency. The reason for the relatively poor performance of the MIL Std. is its longer bus idle time and larger overhead due to its message formating structure. The linear token bus yields a slightly better performance than the token ring, but the difference may not be of statistical significance. ~~

Under each traffic condition and for each protocol the simulation model was run for 2 million ps, i.e., 2 s. Built-in statistical testing routines provided in SIMAN [ 181 were employed to obtain the average values, standard deviations, and confidence intervals at different percentiles of the data latency. Because of the time-varying nature of the network’s operations, these steady-state statistical parameters, except the average values, have no major bearing on the DDCCS network design. In many applications, such as routine information exchange between the design office and factory floor in an autonomous manufacturing environment, message interarrival time T and message length L are random variables. Performance evaluation of the DDCCS network was conducted under four different traffic scenarios: 1) constant T and constant L,i.e., deterministic traffic; 2) constant T and exponentially distributed L; 3) exponentially distributed T and constant L; and 4)both T and L being exponentially distributed. Simulation results for different combinations of constant and exponentially distributed cases of T and L are listed in Table I for the three protocols under consideration. Each protocol was simulated at two values of queue limits: Q = 1 and Q = 2, and three values of offered traffic: G = 0.2,G = 0.7, and G = 1.2. On the basis of Definitions 3 and 4 in Section I1 and the data provided in Appendix C of [ 5 ] , the critical offered traffic G , (see Definition 8) for N = 31 and T = 10 ms was computed to be 0.993,0.986, and 0.523 for the SAE token bus, SAE token ring, and MIL Std. 1553B, respectively. Based on the simulation results provided in Table I, the following inferences can be drawn. 0

For all protocols with G

< G,,, the average data latency

The steady-state performance data, Le., the average data latencies, in Table I are significant from a network design point of view. However, for control system design it is the time-varying characteristics of the delays that also need to be taken into account.

B. Dynamic Performance of the Distributed Control System Since the data latency characteristics of the SAE token ring and linear token bus are similar, to a large extent, under normal operating conditions of the network, the transient responses of the digital control system in Fig. 1 were generated with the linear token-bus and MIL Std. 1553B protocols under different traffic conditions. Transient responses under identical conditions were obtained for an equivalent centralized digital controller, where the control loop is not subject to any delay except the usual sampling time delay. These responses were used as a reference for evaluating the individual protocols in the DDCCS.

903


--- _ _ _ _MIL-STD-15538, 0-2 0-1 _ - _ _ - - - - MIL-STD-15538, SAB Linear Token Bum, 0-1 and 0-2 Reference

Y

g a

0

0.0

0.2

0.4

0.6

0.0

1 .o

Time (sec)

Fig. 3. Transient response of plant output at G

For simultaneous traffic (see Definition 2 in Section 11) in all terminals (except the controller terminal 2), transient responses of the output were obtained for a unit step increase in the reference input from an initial steady-state condition. In real applications the terminals may not be synchronized, and some of them may generate random traffic; therefore the traffic would not be simultaneous. However, simulation results with simultaneous traffic allow a better insight of the characteristics of network-induced delays and their impact on the control system performance. A typical scenario is discussed below. Fig. 3 shows transient responses for G = 0.7, where G exceeds Gcrfor MIL Std. 1553B only, i.e., some messages are lost due to queue saturation. The performance of the MIL Std. becomes much worse, leading to instability, if Q is increased to 2 for the transmitter buffers of all terminals, including those for the sensor and the controller. The cause of the degradation is an additional queueing delay of 10 ms in each terminal in which a message never arrives at an empty buffer. This phenomenon is analytically derived in Proposition 2 of [5]. Simulation experiments were carried out for random traffic with Poisson arrival and exponentially distributed message lengths in terminals 3 to 31 to investigate the effects of stochastic delays on the dynamic performance of the feedback loop in the DDCCS network. Transient responses for the plant output with random traffic were found to have characteristics similar to those with deterministic traffic. Additional results illustrating the impact of network-induced delays on dynamic performance of aircraft and chemical-plant control systems are reported in earlier publications [5], [6]. The overshoot and settling time of transients for the SAE linear token bus are, in general, superior to those for MIL Std. 1553B under identical network traffic, because MIL Std.

=

0.7.

1553B suffers from larger data latencies due to a larger message overhead and idle time. This also entails a relatively smaller critical offered traffic for MIL Std. 1553B, resulting in a larger number of lost messages whenever offered traffic exceeds its critical value. The preceding observations are generic in nature and are applicable to DDCCS networks for power and chemical plants, autonomous manufacturing processes, and aircraft. Although the process dynamics in different applications may vary widely, the concept of dimensionless offered traffic and the resulting delays relative to the sampling period are similar in all cases. V. CONCLUSION

The paper presents the results of an ongoing research in the area of analysis and design for real-time distributed data communication and control systems (DDCCS) networks. This is an evolving field which requires interaction between the disciplines of communication systems and control systems engineering. In this respect, definitions of pertinent DDCCS parameters are introduced and propositions for system analysis have been developed and proved. A generic methodology that builds upon the concept of combined discrete-event and continuous-time simulation has been developed for performance evaluation of real-time distributed control system networks for autonomous manufacturing, electric power and chemical plant, and aerospace applications. The simulation program is modular in the sense that diverse network protocol models and dynamic models of the plant and controller can be incorporated within the program without any need for modifying its structure. The impact of delays induced by different network access protocols, such as the SAE linear token-bus and MIL Std.

904


1553B protocols, on the dynamic performance of a control system has been investigated. The following conclusions regarding DDCCS network design have been derived on the basis of analysis and simulation. Although integrated control systems that are interconnected by a data communication network provide for operational flexibility and evolutionary design, networkinduced delays degrade the system’s dynamic performance and are a source of potential instability. The network-induced delays are dependent on both traffic type and the access control protocol. The control system in general should have a better performance if a distributed controlled-access protocol, such as the SAE linear token bus or token ring, is used instead of a centralized controlled-access protocol like the MIL Std. 1553B. However, this conclusion is made only on the basis of data latency in which failure modes and reliability issues have not been addressed. The network-induced delays are time-varying and possibly stochastic. Conventional frequency-domain techniques that are routinely applied for linear time-invariant systems may not be valid for DDCCS design. Furthermore the delays from sensor to controller and f.rom controller to actuator in a feedback control loop may not be lumped together in general. The network should be designed such that the offered traffic does not exceed the critical value. This implies that any combination of network design parameters (namely, the number of terminals on the network, message arrival rate, message length, medium bandwidth, and sample time) should allow for a safe margin between the offered traffic and its critical value. ACKNOWLEDGMENT The author acknowledges technical contributions of Dr. Y. Halevi, Mr. S. Lee, Mr. S. H. Hong, and’Dr. P. J. Egbelu. F

REFERENCES N. G. Ordrey, “Integrating the automated factory,” in Data Processing Management. New York: Auerback Publishers, Inc., 1985.

T. H. Tsai, J. W. Lane, and C. S. Lin, Modern Control Techniques for the Processing Industries. New York: Marcel Dekker, 1986. A. L. Hopkins et a/., “System data communication structures for active-control transport aircraft,” vol. 11. NASA report CR 165774, June 1981.

A. Ray, R. Harper, and J. H. Lala, “Damage-tolerant multiplex bus concepts for far-term digital flight control systems,” C.S. Draper Laboratory Report CSDA-R-1690, Feb. 1984. A Ray, “Performance evaluation of medium access control protocols for distributed digital avionics,” J. Dynamic Syst. Measurement, Contr., pp. 370-377, Dec. 1987. A. Ray, “Distributed data communication networks for real-time control,” Chem. Eng. Commun., vol. 65, pp. 139-154, 1988. W. L. Hoberecht, “A layered network protocol for packet-voice and data integration,” IEEE J. Selected Areas Commun., vol. SAC-I, no. 6, Dec. 1983, pp. 1006-1013. IEEE J. Selected Areas Commun., Special Issue on Network Performance and Evaluation, A. Leon-Garcia, Ed., vol. S A C 4 , no. 6, Sept. 1986. Y. Halevi and A. Ray, “Integrated communication and control systems: Part I-Analysis,” J. Dynamic Syst. Measurement Contr., scheduled for Dec. 1988. A. Ray and Y. Halevi, “Integrated communication and control systems: Part 11-Design considerations,” J. Dynamic Syst. Measurement Contr., scheduled for Dec. 1988. W. Stallings, Data and Computer Communications. New York: Macmillan 1985. SAE Linear Token Passing Multiplex Data Bus. AS4074.1, version 3.0, May 1987. SAE High Speed Ring Bus Standard. Issue 1, Nov. 1985. Token Passing Bus Access Method and Physical Layer Specifications. IEEE Std 802.4, 1985. Token Ring Access Method and Physical Layer Specifications. IEEE Std 802.5, 1985. MIL-STD-I553B Protocol Manual. U.S. Govt. Printing Office, Feb. 1980. J. Banks and J. S. Carson, “Process-interaction simulation languages,” Simulation, vol. 44, no. 5, pp. 225-235, May 1985. C. D. Pegden, Introduction to SIMAN. State College, PA: Systems Modeling Corporation, 1985.

Asok Ray (SM’83) received the Ph.D. degree in mechanical engineering (Northeastern University, 1976) and the master’s degrees in electrical engineering, computer science, and mathematics. He has about ten years of research and management experience at GTE Strategic Systems Division, Charles Stark Draper Laboratory, and the MITRE Corporation. He has also held research and academic positions at Carnegie-Mellon University and the Massachusetts Institute of Technology. He joined the faculty of mechanical engineering at The Pennsylvania State University as an Associate Professor in July 1985. His research experience includes real-time microcomputer-based distributed control and instrumentation, networking and communication protocols, intelligent systems design, and modeling and simulation of dynamical systems as applied to avionics, process control, and autonomous manufacturing. His current research interests include distributed control systems, knowledge-based expert systems, robotics, and communication networks. He has authored or coauthored over 100 research publications. Dr. Ray is a senior member of the IEEE and a member of the ASME.