Enhancements to the Time Synchronization Standard IEEE-1588 for a System of Cascaded Bridges J¨urgen Jasperneite Phoenix Contact GmbH & Co. KG Dringenauer Strasse 30 31812 Bad Pyrmont, Germany
[email protected]
Khaled Shehab University of Paderborn Warburger Str. 100 33098 Paderborn, Germany
[email protected]
Karl Weber Siemens AG W¨urzburger Strasse 121 90766 Fuerth, Germany
[email protected]
Abstract The IEEE-1588 standard for a high precision time synchronization now exists since 2002. For using this standard in bridged networks a so-called boundary clock is defined, where the local clock adjustment can be modeled by a corresponding control loop. At the field level of industrial automation systems, the line topology is very important. By using Ethernet at the field level, the resulting chain of bridges leads to a cascade of control loops and may lead to instabilities and deviations of the distributed clocks, which are not acceptable. For this application a bypass clock instead of the boundary clock is proposed as an enhancement of the IEEE-1588 standard. The effectiveness of this extension will be evaluated by simulation technique.
1 Introduction For many applications Ethernet becomes an alternative to fieldbus systems. This due to a decrease in price provoked by the office Ethernet market, high bandwidth, switching technology [3], priority features [4], full duplex operation [2], availability of Ethernet bridges as well as Ethernet-enabled products fulfilling industrial environmental requirements (e.g. [15]). The term bridge is used according to ISO/IEC 8802-3 and is synonym to switch [5]. Bridges are used because they can connect different network technologies at MAC level with high throughput and little configuration effort. They provide means for traffic reduction by forwarding frames selectively while repeater technologies will flood all attached network segments (collision domain) with every single frame. Thus bridges can be used in networks with a high number of
stations with little performance degradation. This basic bridge feature allows to be independent of the network size. At the field level of industrial automation systems, the line topology is very important [16]. Wiring follows cabling channels that often lead to a structure of a line with some trunks. Thus, an industrial control system may have tens or even up to hundreds of bridges cascaded. Especially in a chain of bridges, the latency and the jitter for real-time data transfer will be increased significantly [17]. Today, latency and jitter lead to restrictions when using standard Ethernet in time-critical synchronous applications, e.g. Motion Control, with a maximum latency in the range of 1 ms and lower and an allowed jitter of below 1 µs. To use Ethernet also for these applications, modifications at the MAC level are necessary, which are often based on the TDMA (time division multiple access) mechanism. In TDMA, pre-allocated time slots are defined for the transmission of time-critical and non realtime data (e.g. TCP/IP). For an overview of the manufacturer concepts discussed at present see [7]. For TDMA schemes like the one used in the isosynchronous version of PROFINET [1], a notion of global time at the nodes and bridges is necessary for packet scheduling. Therefore the distributed clocks in the nodes and bridges must be synchronized very precisely. The protocols used for synchronization and scheduling should be designed to make full use of the highest possible speed even if a part of the system uses links of lower performance (e.g. wireless links). The time of a client relative to its server can be expressed by T (t) = T (t0 ) + R(t − t0 ) +
D(t − t0 )2 2
where t is the current time, T is the time offset at the last
measurement update t0 , R is the frequency offset and D is the drift due to resonator ageing, temperature change or mechanical shock [12]. All three terms include systematic offsets that can be corrected and random variations that cannot. Not all sources for systematic offsets can be corrected (e.g. asymmetry of line length ...), but this is beyond the scope of the paper. Some protocols estimate only the first term in this expression, while others, including NTP [11], estimate the first two terms. Errors due to the third term, while important to model resonator aging in precision applications, are neglected, since they are usually dominated by errors in the first two terms. A synchronization protocol estimates T (t0 ) (and R(t0 ), where relevant) at regular intervals t and adjusts the clock to minimize T (t) in future. In common cases, R can have systematic offsets of several hundred parts-per-million (PPM) with random variations of several PPM due to ambient temperature changes. If not corrected, the resulting errors can accumulate to seconds per day. In order that these errors do not exceed a nominal specification, the protocol must periodically reestimate T and R and compensate for variations by adjusting the clock at regular intervals. For our desired kind of applications with nominal accuracies of microseconds, this requires clients to exchange messages with servers at intervals in the order of seconds or below. A emerging standard for this accuracy is the Precision Time Protocol (PTP), also known as IEEE-1588 [6, 8].
the time difference between master and slave is corrected. During this offset correction, the master cyclically transmits a Sync Message containing an estimated value (a0 ) for the exact time the message was transmitted (T1 ) to the related slave at regular intervals (Sync interval). In the Follow up Message the master sends the more precise measured value of the time (T1 ) the Sync message was placed on the network. In the second step, the delay measurement determines the propagation delay between slave and master. For this purpose, the slave sends a so called Delay Req Message to the master and gets back a Delay Resp Message. The slave uses this to calculate precisely how long it takes to transmit the message over a particular network section. After these two steps, the slave is now able to calculate the propagation delay and the offset: A = T2 − T1 = Delay + Of f set B = T4 − T3 = Delay − Of f set Delay =
(A − B) 2 Delays at the transmission channels are fixed and several hundreds of nanoseconds for maximum extension of a point-to-point network (e.g. 500 ns for 100 m STP). They can be compensated if they are symmetrical. Drifts are mainly caused by deviation of the frequencies of the clock sender from the frequency of the clock receiver. They are mainly caused by temperature and ageing. As temperature change may occur in various kinds they should be compensated by a control loop. Each PTP slave and thus each slave port of each boundary clock must have such kind of loop. The loop design is beyond the scope of PTP. But their design determines the quality of time synchronization and should be defined for the clock design within bridges [13]. It is of less importance for end nodes, which have no networking purpose. In this paper we choose a simple PI-Loop (Type II) to adjust our local clock (Fig. 2) with the following discrete filter function [9]: Of f set =
2 Brief description of IEEE-1588 (PTP) PTP is based on a master/slave procedure. The synchronization of the clocks will be realized by special synchronization telegrams. Like most well-known time synchronization procedures, including NTP [11, 10]), PTP is based on a T1 − T4 transaction model (see Fig. 1). The synchronization process is carried out in two steps, the offset and the propagation delay measurements. First 0DVWHU D
7
6ODYH 6\QFB0VJ>D@ )ROORZBXS>7@
'HOD\B5HT 7
(A + B) 2
y[k] = y[k − 1] + KR ((1 + 7 7 'HOD\2IIVHW
T )x[k] − x[k − 1]) (1) Tn
where y[k] is the manipulated variable and x[k] is the error variable at synchronization cycle k, y[k − 1] and x[k − 1] the corresponding values of the last synchronization cycle k − 1 and KR , Tn are the control loop parameters. T is the sampling time, which is equal to the Sync interval of PTP.
7
'HOD\B5HVS>7@
3 The Boundary Clock 7 7 'HOD\2IIVHW
A boundary clock (BC) is an IEEE 1588 component that allows the synchronization of IEEE 1588 clocks across subnets defined by a router or bridges that blocks
Figure 1. Transaction model of PTP 2
5HIBFORFN
[
(UURUIXQFWLRQ /RFDOFORFN
3,)LOWHU
\ 373 0DVWHU
9&2
373 %ULGJH
7LPH6HUYHU UHIHUHQFHFORFN
Figure 2. Local clock correction with a PI control loop
373 %ULGJH
373 %ULGJH
373 6ODYH
7LPH&OLHQW 373 6ODYH
373 PDVWHU
2VF FORFN FRUUHFWLRQ
the transmission of all PTP messages (see top of Fig.3). A BC serves to eliminate the large fluctuations in communication latency typically generated by bridges. To realize time synchronization with PTP for a line topology the BC must be used. The principle of synchronization through a sequence of PTP Bridges can be seen in Figure 3. The node that originally places the time frames for synchronization in the network is known as the PTP master. All other nodes that receive and/or pass on these frames are known as PTP bridge or PTP slave. A PTP master uses a reference to a local time system or a global time source. As described in IEEE 1588, the BC can have either all its ports in the PTP master state or only one of its ports in the PTP slave state and the rest are in PTP master state. In this paper we took the 2nd case. The oscillator (Osc.) of the local clock within a BC will be regulated by the PTP slave functionality. The corrected local clock of the BC will be used as the master clock for the next network segment. This procedure will be repeated until reaching the time client. In this way a cascade of control loops will be generated. In a chain of bridges this may lead to instabilities and deviations of the distributed clocks, which are not acceptable. Because of the importance of the line topology at the field level of industrial automation systems it is necessary to examine a further synchronization element beside the already standardised IEEE-1588 synchronization elements, which we call a Bypass Clock (BpC).
%RXQGDU\&ORFN %&
373 %ULGJH
373 0DVWHU
7LPH6HUYHU UHIHUHQFHFORFN
373 %ULGJH
373 %ULGJH
373 6ODYH
7LPH&OLHQW PHVVDJH SURFHVVLQJ 2VF FORFN FRUUHFWLRQ %\SDVV&ORFN %S&
Figure 3. Boundary Clocks (BC) and Bypass Clocks (BpC) in a Line Topology
the value of the individual jitters, but not from the control loop design and the drift in the bridges. A drift compensation within brigdes may be done if the delay time of a single bridge is substantial (in the range of 1 ms or above). The jitter may vary from bridge to bridge and the probability to detect extreme deviations may be smaller than the number of nodes increases. Thus the absolute error is larger for high cascaded systems but the worst case factor shall be smaller.
4 The Bypass Clock
OG
As mentioned before, the basic issue with bridges is the non constant delay. The delay shall be known at the PTP slave. Otherwise the precision depends on the way the delay changes. This delay could be compensated if it is calculated by the PTP bridges. The BpC (see bottom of Fig. 3) compensates the latency caused by the bridges by manipulating the appropriate PTP telegrams. The paper does not want to show how this manipulation is accomplished. Adding the delay between receipt and sending a Sync message will provide a means to treat bridges like network components with constant delay. With this approach, no control loop is needed for forwarding Sync messages. The precision of this approach depends upon
373 0DVWHU
EG
OG
373 %ULGJH
EG
OGQ
373 %ULGJH
373 6ODYH
'6\QF
Figure 4. Delay accumulation If a PTP bridge receives a Sync message on port s, the entry timestamp Trx,s and the corresponding information (sequence number, etc.) from the Sync message will be extracted. It is essential to know on which port s the par3
1 bridge 6 4 abs. error [us]
0.2 0.0 −0.2
0 −2 −4 −6
0
• and the line delay ldi to the previous node (for line delay measurement)(see Fig. 4).
i=1
2
−0.4
• adding its internal bridge delay bdi for handling time frames
20
40
60
80
0
40
60
80
simulation time [s]
5 bridges
10 bridges 150 100 abs. error [us]
10 0 −10
50 0 −50 −100 −150
−20
i=1
20
simulation time [s]
20
abs. error [us]
The cumulated delay of a Sync-message DSync through the network from the PTP master to the PTP slave is given by: n n−1 X X DSync = ldi + bdi
3 bridges
0.4
abs. error [us]
ent is located to assign the appropriate Follow up frame. All j ports of the Bypass Clock forward every received Sync message, except for the s port of the bridge that initially received the Sync message. For each port, at which the Sync message is sent, the timestamp unit generates the respective time stamp Ttx,j . The required correction is Ttx,j − Trx,s . In principle the Sync frames are forwarded like other frames. In addition, the time had to be adjusted in every each bridge by
0
with n as the number of links between the PTP master and the PTP slave. The main difference to the scenario with boundary clocks is that in the case of bypass clocks only one control loop between the PTP master and the PTP slave exists.
20
40
60
80
simulation time [s]
0
20
40
60
80
100
simulation time [s]
Figure 5. Absolute error ε of clock synchronization for different number of cascaded bridges using boundary clocks
5 Simulation Study 10 ns. Clocks other than the grandmaster drift from the grandmaster clock with an assumed rate of +1µs/s. A PI-loop of type II corresponding to Figure 2 with a filter function according to Equation 1 is used to compensate the drift of the local clocks. Appropriate values were taken for P and I factors to avoid unstable conditions. P = 1 and I = 0.25 seem to be appropriate for a controlled system with unknown behaviour. Each bridge is synchronized to the bridge that precedes it and synchronizes the bridge that follows it. A full duplex channel (FDX) with a transmission capacity of C = 100 Mbps/s is assumed. The propagation delay between any two nodes is 20 ns and is assumed to be symmetrical in both ways. A hardware time-stamping very close to the transmission medium was assumed in the model. Concerning the IEEE 1588 standard, a Sync interval of 1 s is chosen. Each Sync-Message is followed by a Follow up-Message. Starting to send Sync- and Follow upMessages is assumed to be random and within the time intervals as specified in the standard. The time interval for sending the Sync-Message is given by
In the following experiment we want to examine first the characteristic of the Boundary clock during cascading. Subsequently, the Bypass clock is to be tested under the same conditions. The goal of this investigation in this paper is to compare these two architectural synchronization elements. For simulation purposes, we use the object-oriented discrete time event simulation environment OMNeT++, which is free of charge for teaching and research use [14]. The appropriate models of the system had to be developed in OMNeT++ according to the valid standards and specifications. A line topology corresponding to Figure 3 was chosen to study the effect of cascading bridges on the accuracy of clock synchronization. The synchronization accuracy is evaluated through calculating the absolute error ε between the Time client and the reference clock (Time master) of the system. ε = TT ime client − TT ime server Grandmaster clock is another designation for reference clock. The grandmaster clock is the most accurate clock in the system and is assumed to be the time base to which all nodes in the system can synchronize their clocks with a high degree of precision. All clocks operate at a frequency of f = 100 Mbps, so that we get a clock granularity of
[0,
Sync interval ] 2 · P T P Random Slot
and for the Follow up-Message by [
4
Sync interval 1.5 · Sync interval , ] (2 · P T P Random Slot) (P T P Random Slot)
1 bridge
There is no traffic in the network other than PTP messages, because we are only interested in a qualitative comparison of the two architectural synchronization elements.
0.6
0.4
0.4
0.2 0.0 −0.2
0.0 −0.2 −0.4
−0.6
−0.6 20
40
60
80
0
20
40
60
simulation time [s]
simulation time [s]
5 bridges
10 bridges
0.6
0.6
0.4
0.4
0.2 0.0 −0.2 −0.4
80
0.2 0.0 −0.2 −0.4
−0.6
−0.6 0
20
40
60
80
simulation time [s]
Figure 5 shows the results of switch cascading with boundary clock implementation. As could be seen, with one bridge between the sender and receiver clock the maximum absolute error ε is about 0.4 µs as a result of the drift correction process in the clock using PI-control loop with the selected parameters described above. After the first synchronization interval at t = 1s the clock that is near the grandmaster will feel that its clock operate at a different frequency (100, 000, 100 Hz) than the grandmaster (100, 000, 000 Hz) and starts to correct its frequency to that of the grandmaster.
0.2
−0.4
abs. error [us]
Four simulation runs were conducted with different number of cascaded bridges between the sender and receiver being 1 bridge in the 1st run, 3 bridges in the 2nd run, 5 bridges in the 3rd run, and 10 bridges in the 4th run.
0.6
0
abs. error [us]
Initially all clocks were assumed to be synchronized (i.e ε = 0), but they have a drift rate of 1 µs/s. After a simulation time of t = 50s a frequency step of 100 Hz at the grandmaster clock took place to observe the response of the system. Other types of random errors could be used, without affecting the result qualitatively.
3 bridges
abs. error [us]
abs. error [us]
We took the default value of the PTP Random Slot to 18 . So the Sync-Message should be sent within a time interval of [0, 0.02777] sec. and the Follow up-Message within [0.02777, 0.08332] sec. We used a uniform distribution function (which is available in OMNeT++ ) to generate random numbers for sending the Sync- and Follow up-Message out of the above specified time intervals.
0
20
40
60
80
simulation time [s]
Figure 6. Absolute error ε of clock synchronization for different number of cascaded bridges using bypass clocks
6
Conclusions
This paper studies the effect of cascading bridges on the accuracy of clock synchronization. To use Ethernet also for hard-realtime applications, modifications at the MAC level are necessary, which are often based on the TDMA (time division multiple access) mechanism to distinguish between the transmission of time-critical and non realtime data. For this an precise global time base is necessary in the network components and the devices. The Precision Time Protocol (PTP), also known as IEEE-1588 standard, adresses this class of applications. At the field level of industrial automation systems, wiring follows cabling channels that often lead to a structure of a line with some trunks. Thus, an industrial control system may have tens or even up to hundreds of bridges cascaded. In our simulation study we could show that it can come with the use of the Boundary clocks as a IEEE-1588 synchronization element to inaccuracies, which are not acceptable. The reason lies in the cascading of control loops between the Reference clock and the Time client. Because of the importance of the line topology at the field level of industrial automation systems it is necessary to introduce a further synchronization element beside the already standardised IEEE-1588 synchronization ele-
It is clear that synchronization works well with one control loop between the PTP master and PTP slave. When cascading two or more control loops clock synchronization starts to degrade. This can be seen by observing the maximum absolute error if more control loops are cascaded. For example, when cascading five bridges (or control loops) the maximum absolute error is about 20 µs and reaches about 150 µs in the case of cascading ten bridges. This result shows that the Boundary Clock is unsuitable for a precise clock synchronization in a line topology with its implicit chain of bridges. As mentioned above, with a Bypass clock only one control loop exists between the PTP master (in this case the system grandmaster) and the PTP slave. Figure 6 shows that the maximum absolute error is well below 1µs even when cascading more than one bridge. This results shows that during the use of Bypass Clocks the deviation between the local clock and the reference clock is almost independent of the number of cascaded bridges. It is apparent that the synchronization quality is improved when the bypass clock implementation is used in such an application. 5
ments, which we call a Bypass Clock. A Bypass Clock only forwards the Sync messages and puts the local bridge delay and the line delay inside this message. In this way, the end node then receives a precise delay of the messages. The main difference to the scenario with Boundary clocks is that in the case of Bypass Clocks only one control loop between the Reference clock and the Time client exists. The simulation experiment could show that during the use of Bypass Clocks the deviation between the local clock and the reference clock is almost independent of the number of cascaded bridges. The Bypass clock introduced here is superior to the Boundary clock for precise clock synchronisation in a system of cascaded bridges. Further investigations, however, must take the next step by replacing the simplified assumptions of system parameters of our simulation model with more realistic values.
[14] [15] [16]
[17]
References [1] IEC. Digital data communications for measurement and control - Profiles for ISO/IEC 8802-3 based communication networks in real time applications MT9 / WG1, 2003. new work item proposal. [2] IEEE. Specification for 802.3 Full Duplex Operation. IEEE, New York, 1997. ANSI/IEEE Std. 802.3x. [3] IEEE. Media Access Control (MAC) Bridges. IEEE, New York, 1998. ANSI/IEEE Std 802.1D-1998. [4] IEEE. Virtual Bridged Local Area Networks. IEEE, New York, 1998. ANSI/IEEE Std 802.1Q-1998. [5] IEEE. Carrier sense multiple access with collision detection (CSMA/CD) access method and physical layer specifications. IEEE, New York, 2000. ANSI/IEEE Std 802.32000 Edition (ISO/IEC 8802-3:2000(E)). [6] IEEE. IEEE Standard for a Precision Clock Synchronization Protocol for Networked Measurement and Control Systems. IEEE, New York, 2002. ANSI/IEEE Std 1588-2002. [7] J. Jasperneite and P. Neumann. How to guarantee realtime behavior using ehernet. In 11th IFAC Symposium on Information Control Problems in Manufacturing (INCOM2004), Salvador-Bahia, Brazil, April 2004. [8] Kang, L. and Eidson, J., editor. Workshop on IEEE-1588, Standard for a Precision Clock Synchronization Protocol for Networked Measurements and Control Systems. National Insitute of Standards and Technology (NIST), Sept. 2003. [9] H. Lutz and W. Wendt. Taschenbuch der Regelungstechnik. Verlag Harri Deutsch, Frankfurt/Main, 1998. ISBN 3-8171-1552-0. [10] D. Mills. Precision synchronization of computer network clocks. citeseer.nj.nec.com/ mills94precision.html, 1994. [11] D. L. Mills. Internet time synchronization: The network time protocol. Network Working Group Request for Comments: 1129, oct 1989. [12] D. L. Mills. Executive summary - computer network time synchronization. http://www.eecis.udel.edu/ ˜mills/exec.html#intro, 2003. [13] M. Mueller and K. Weber. Impact of Switch Cascading on Time Accuracy. In Workshop on IEEE-1588, Standard for
6
a Precision Clock Synchronization Protocol for Networked Measurements and Control Systems. National Insitute of Standards and Technology (NIST), September 2003. OMNeT. OMNeT++ Discrete Event Simulation System. www.omnetpp.org, 2003. Phoenix Contact. Industrial Ethernet Products. www. ethernetrail.com, 2003. S. R¨uping, E. Vonnahme, and J. Jasperneite. Analysis of Switched Ethernet Networks with different Topologies used in Automation Systems. In D. Diedrich, P. Neumann, and H. Schweinzer, editors, Field Bus Technology, pages 351–358. Springer-Verlag, 1999. K. Watson and J. Jasperneite. Determining end-to-enddelays using network calculus. In 5th IFAC International Conference on Fieldbus Systems and their Applications (FET’2003), pages 255–260, Aveiro, Portugal, July 2003.
