clocks in a discrete event simulation environment. I. INTRODUCTION ... an emergency stop message is delivered too late or even not at all. Despite those ...
Distributed Clock Synchronization in Discrete Event Simulators for Wireless Factory Automation Felix Ring, Georg Gaderer, Anetta Nagy, Patrick Loschmidt Institute for Integrated Sensor Systems Austrian Academy of Sciences 2700 Wiener Neustadt, Austria Email: {Felix.Ring, Georg.Gaderer, Aneta.Nad, Patrick.Loschmidt}@OEAW.ac.at
Abstract—One very interesting trend in factory automation is the introduction of wireless networks to the factory floor. This requires investigation of problems which are not present with wired technology. Tracking of mobile nodes, as well as seamless handover are such problems. Precise clock synchronization is needed to solve these new issues. For large-scale systems it is economically unreasonable to implement complete factory scenarios with prototypes in order to test for architectural, engineering, and design problems. It is mandatory to have a proper simulation environment at hand in order to study the influence of various design decisions. This paper discusses techniques of how to simulate such a large-scale system with special focus on proper modeling and simulation of distributed clocks in a discrete event simulation environment.
I. I NTRODUCTION One of the currently arising trends in factory automation is the use of wireless technologies on the shop floor. Their introduction is not as simple as in the home or office domain. This is due to the fact that one can consider wireless connectivity in the office world as a best-effort service, although for applications like voice-over-IP or video-streaming some guarantees have to be given. These restrictions are, however, relatively relaxed. For example, during a short phase of communication in the voice-over-IP application, packets can be either stored and forwarded later, or even dropped with the disadvantage of a degradation of audio quality. Even more, one can argue that these soft real-time guarantees can be largely relaxed by instantiation of higher bandwidth capacities.1 In industrial environments harder real-time guarantees need to be met. Missing a deadline in control networks can lead to a variety of more or less severe consequences, starting from a degraded production quality due to the loss of sampling points for control loops up to life threatening situations if an emergency stop message is delivered too late or even not at all. Despite those challenges, wireless technology in the automation domain enables a whole new set of applications in terms of mobility and flexibility. Although several technologies for cable replacement like Bluetooth® already exist, the problem for the scope of wireless 1 This project was partly financed by the province of Lower Austria, the European Regional Development Fund, the FIT-IT project ε-WiFi under contract 813310, the EU under the FP7 STREP flex WARE contract number 224350 as well as the Austrian Center of Competence in Mechatronics (ACCM) under contract A260501 and Oregano Systems Design and Consulting.
automation of a whole factory floor is not yet solved. In this deployment area a potentially large number of nodes need to be coordinated. The use of hardware prototypes for design and development of such a large-scale system is not feasible, due to the large amount of required nodes. Thus, a proper simulation framework has to be developed. This simulation framework has to provide all the specific properties of the system under consideration. For a meaningful and detailed simulation it is required to properly model the time base of all the independent local nodes. This paper proposes such a framework for a system, where a very fine-grained real-time enabled time base is used for features like localization and hard real-time support. After a short introduction to the concept of flexible wireless automation, the general system concept will be presented and developed. The concepts are shown in the specific use-case of a simulation created for the flex WARE project [1], which aims at developing a complete system for wireless real-time communication on the factory floor. The remaining part of the paper discusses the implementation and presents preliminary results of the system. Finally, some outlook and conclusions will be given. II. F LEXIBLE W IRELESS FACTORY AUTOMATION Evolution in factory communication started with the introduction of fieldbuses: They were step-by-step replaced by the nowadays standard Ethernet real-time technologies. Similarities to the deployment of wireless networks arise [2]. The introduction of Ethernet was not straight forward, as the task at this time was to use a primarily non realtime enabled technology for the demanding hard real-time factory automation. Nevertheless, the challenges were taken and solutions were developed, not only to gain the advantages of Ethernet chipsets being available on the mass market, but also to have the same means of communication from the factory management down to the shop floor. The aimed usage of wireless communication in factories presents a similar scenario. For cost-effectiveness it is desirable to use as much consumer grade hardware as possible, but additional problems need to be tackled, which are not known in wired real-time networks: • Mobility is inherent to wireless nodes. This introduces the problem that traditional topologies can not be used
•
•
any more. Nowadays, wired automation devices are typically using topologies like lines or open rings. This has the advantage that the schedulability analysis for realtime communication can be done statically in advance, which makes it easy to calculate and provide real-time guarantees. In wireless systems this is no more true. A factory-wide wireless infrastructure has to cope with the problem of dynamic topologies, and thus has to schedule dynamically when the position of a node changes (e. g., because a node needs to be associated with another access point due to bad link quality of the current connection). Localization of mobile nodes is mandatory for two reasons. Firstly, the system itself needs to know about the position of the mobile devices in order to predict a change of the access point reception for the above mentioned dynamic schedulability analysis. Secondly, several applications can consider the position of nodes for safety reasons, e. g. direct line of sight is required to operate machines, or collision avoidance mechanisms for automated guided vehicles. Scalability is a third new aspect in wireless real-time systems. The concept of gateways and multicast domains is very well established for Ethernet, as the network topology is known a-priori. This is no longer the case in the wireless domain, since the system has to be able to dynamically adapt and grow. Nodes can be added at any arbitrary place within the coverage area without having to reconfigure the complete system. This growth and adaptation is needed even in the simple case of devices being turned on, or when they loose and regain connection during runtime. III. S YSTEM C ONCEPT
To tackle those challenges, a complete view of the system in terms of location and resource usage is needed. Therefore, a central monitoring and controlling device, called the flex WARE controller, is used. For the sake of scalability, the system allows to have several controllers in one installation in order to balance the communication load. Details on the system concept can be found in [3]. A.
flex
WARE System
The structure of the system is shown in Fig. 1. At a very early point of the project it was decided to use wireless LAN (IEEE 802.11) as a communication technology, in order to take advantage of the high volumes of consumer-grade chipsets. As shown, the system consists of four distinct main components: 1) Automation Device: Those components are standard application devices like sensors, actuators, AGVs, conveyor belts etc. From the flex WARE communication system’s point of view they represent the data sources and sinks. 2) flex WARE Node (FN): The FN is the endpoint of the flex WARE communication system. It connects to the automation devices via an integrated interface to the implemented fieldbus. It is planned to equip these elements as far as possible with Commercial-Off-The-Shelf (COTS) chipsets.
Controller (4)
Access Point (3)
Node (2)
Automation Device (1) Fig. 1.
System concept and main components.
As the localization subsystem does not require any kind of timestamping at these entities, the hardware for the mobile devices can be held simple. This is important to reduce production costs, since the nodes presumably will have the highest volume. For real-time communication the nodes still need to be synchronized. Measurements for achievable synchronization results with COTS hardware are presented in [4]. Corresponding results of the simulation are presented later in this paper. 3) flex WARE Access Point (FAP): The FAP mainly acts as a bridge between the wired and the wireless medium. It is enhanced with additional functionalities, like the scheduling of aperiodic real-time messages. In the actual implementation the communication between FAP and FC will be established using IEEE 802.1q capable Ethernet links. As the FAPs are used as reference points for the Time Difference of Arrival (TDoA) based localization subsystem [5], they need to be capable of high-precision clock synchronization with an accuracy down to several nanoseconds. Therefore, hardware timestampers are used in the FAPs. How to properly model and simulate a synchronization system of such high accuracy is discussed in this paper. 4) flex WARE Controller (FC): The task of the controller is to schedule all traffic in its domain. It is also in charge of tracking the location of the nodes in order to prepare a dynamic re-scheduling to maintain real-time guarantees. As the actual behavior in terms of spatial distribution of the nodes is unpredictable, the FC is also in charge of monitoring the available resources in order to generate a warning if the resources, e. g. communication bandwidth for one cell, is reaching its limits, and to redistribute and reschedule the current load, if necessary and feasible. From the applications point of view, communication over the flex WARE system is transparent. A network planning tool configures the controller in a way that the automation devices are virtually connected via an Ethernet link. In fact, this link is transparently routed by the controller. From a system point
simulation implementation of an attached fieldbus. Instead, a traffic generator is used, which simulates the behavior of the attached application system. Due to the modularity of the simulation framework the traffic generator can easily be replaced by any existing or newly developed implementation of the respective fieldbus system. C. Simulation Tool
Fig. 2.
Simulation model structure of the
flex WARE
node
of view, flex WARE is intended to be a protocol transparent, real-time enabled cable substitute for any kind of enclosing communication system. The communication requirements for these virtual links (which can also have several endpoints), are defined by a set of real-time requirements (latency, bandwidth, deadlines, etc.), which are known before runtime. B. Simulation Concept To be able to simulate the flex WARE system, each of the previously described main flex WARE components has its representation as a simulation module. The simulation modules consist of further submodules, which implement models of either physical or logical aspects of the main unit. Fig. 2 shows the composition of the flex WARE node, with the submodules involved in clock synchronization being highlighted. In the simulation, only a single FC is implemented. Since flex WARE gives no real-time guarantees for roaming from one FC domain to another, two distinct FC domains can be viewed as separate functional domains, hence it is not necessary to simulate handover between those domains. The simulation setup further contains multiple stationary access points, as well as multiple mobile nodes. The access points are connected to the FC with a Gigabit Ethernet link, while the nodes communicate with the access points using the IEEE 802.11 protocol. The simulation of the flex WARE system aims at creating a simulation platform to test and evaluate various flex WARE internal protocols, scheduling algorithms, and mechanisms to overcome the inherent malicious effects wireless communication has on real-time systems. Special emphasis is put on a realistic simulation of system critical procedures like the real-time scheduling, handover, localization, and as a base for all the others, distributed clock synchronization. Because of this primary goal, less emphasis is put on the simulation of the enclosing fieldbus. As described above, the flex WARE system is transparent for the applications communication system. Therefore, it is not necessary to have a full featured
Among the various candidates for a simulation tool for the described system simulation the OMNeT++ [6] simulation environment was chosen. The decision for this tool was based on the experience of other research groups [7], as well as on an investigation of the possibilities for modularization and scalability of various simulation environments. OMNeT++ is a modular, component-oriented C++ based discrete event simulator. It is typically used to simulate computer networks. OMNeT++ provides an open-architecture for developing modules providing the possibility to interconnect them into a compound component or network. The modules interact with each other by exchanging messages, which are treated as events. The scheduling of these events is based on the simulation wide global time base, called the simulation time. This time base is not a continuous time scale, due to the nature of discrete event simulators – it is rather represented by discrete timestamps, which are added as a property to each event. The central event scheduler stores all future events in a sorted list and processes them, earliest occurrence time first. For each event the simulation time is set to the requested occurrence time, and the issuing module is awakened by receiving the scheduled message. Modules can also issue so-called self-messages, which are received by the issuing module at a given future point in time. They can be used as timers, as the module gets activated at the specified time. As this activation mechanism is controlled by the central event scheduler, every module has the same understanding of time, hence, distributed clocks cannot be simulated. IV. I MPLEMENTATION To achieve the simulation of real time in the sense of independent, distributed clocks, a novel concept for eventbased timing is proposed, which is already implemented in the flex WARE framework for OMNeT++ simulations. A. Introducing Real-Time in Simulation – The Local Virtual Time Base Because of the central event scheduler, which is based on the global simulation time, all modules have exactly the same notion of time. Considering the achievable accuracy for wired clock synchronization of about 20 ns this would be a proper simplification for most applications, if not, as in our case for the localization system, clock deviations in the nanosecond area are of interest. Even more for a wireless network scenario, where clock synchronization accuracy is in the microsecond range at best, a proper simulation of the distributed clocks is needed. In order to get an accurate model of reality, it
B. Local Clock Model A clock usually is a two-parted device, consisting of an oscillator and a counter. The oscillator (frequency standard) determines the length of the second, or of a time interval known as the clock increment. The counter keeps track of the number of oscillator ticks and sums up the clock increments between oscillator ticks, hence the name adder-based clock. As the oscillator is the part which causes most of the imperfections of a clock, the credibility of the clock model heavily depends on the oscillator model. To create a realistic model of oscillators, their real-world representatives need to be understood. Nowadays, many types of oscillators are available. The commonly known XOs (Crystal Oscillators) are low-cost AT-cut crystals. Their phase noise is mainly influenced by their frequency and temperature, which are neither physically stabilized nor mathematically compensated. Oscillators are in many ways production and environment dependent. The actual stability of an oscillator is dependent on several external parameters, such as the above mentioned temperature, supply voltage, and mechanical effects like vibration, etc. All of these effects can be summarized in terms of their influence on the frequency stability or equivalent phase stability. Some of them are deterministic (such as the temperature behavior) and others are stochastic. A model for the first kind of influences is published in [8]. The following considerations take a constant set of these parameters into account and, in a first step, only characterize the behavior of an oscillator in a stable environment [9]. The nondeterministic oscillator noise has major impact on the mathematical description of the model. The usual means of statistical description of a process, like the standard deviation, is no longer valid, as the values do not converge. Thus, besides
−5
10 Time error, xi (s)
is necessary to model the imperfections of the individual, distributed clocks in simulation. The proposed new approach uses a technique to accurately simulate a distributed clock model. For each physical unit which comprises a clock in the real system, a local clock module in simulation should be implemented. The local clock module acts as an interface between the simulation-time based message scheduling and the local modules. All modules which are located in the local component, and thus are controlled by the same clock, are not allowed to use the standard timer mechanism provided by the simulation tool, which is the issuing of self-messages. Instead, each module has to register a callback function at the local clock module. The local clock module then becomes a local event scheduler, which (locally) replaces the central event scheduler of the simulation engine. The notion of time at each component comprising its own local clock model can be viewed as a local virtual time base. The time base is local, because it is only valid in the context of the specific component. It is called virtual, because there is no consistent representation of time (e. g. in terms of clock ticks), but only a per event calculated offset to the simulation time. Fig. 5 shows an example measurement of the difference of the local clock to the simulation time.
x 10
5
0
−5
0
1000
2000
3000
4000 5000 Simulation time (s)
6000
7000
8000
9000
Fig. 3. Example of the random walk of a clock. The illustration shows the offset of a clock, which is uniformly sampled and departs from the ideal time with an effect called random walk
the observed stationary noise, also other effects like random walk (a part-time trend of a clock to depart from the reference depicted in Fig. 3) have to be considered. Moreover, the measured values for the clock error highly depend on the sampling interval, i. e. in our case the frequency of our observations. This behavior can be described very efficiently with the power spectral density of the noise, or in the time-domain with the so-called Allan deviation [10]. The effects of these different noise types which have to be taken into consideration for a realistic clock model are typically relatively low. Consequently, in most cases it is sufficient to model the jitter of the oscillator ticks with a standard-deviation-type (stationary) noise. For the system under consideration this is no more true, as the accuracies which are needed for localization require clock synchronization in the nanosecond range. Details on modeling this behavior are given in [11]. C. Synchronization in Simulation In a simulation with a realistic, distributed clock model the clock synchronization mechanism needs to be simulated as well to get an accurate and meaningful representation of the real system. In Fig. 2 the simulation modules which participate in clock synchronization are highlighted. 1) Oscillator Model: The local frequency standard of a component. 2) Precision Clock: The local adder-based clock with a variable time increment, which is controlled by a PI-type control loop. 3) Timestamper: The timestamping unit represents a hardware or software timestamper, which timestamps the event messages defined in the IEEE 1588 standard [12]. A physical timestamper listening on the Media Independent Interface (MII) between the PHY and the MAC has the advantage that it does not experience the processing jitter of the upper layers, introduced by the operating system, nor is it affected by the transmission delay on the MAC layer due to the medium access scheme. The only remaining uncertainty is the one introduced by the physical layer. 4) PTP Stack: The local clock module implements the Precision Time Protocol (PTP), which enables clock synchronization between the master clock and the slave clocks. The local adder-based clock can be controlled in state (resetting
−9
−6
10
2
Measured Modeled
x 10
−8
10
1
−10
0 Local time offset (s)
Allan variance
10
−12
10
−14
10
−1
−2
−16
10
−3 −18
10
−4 −20
10
−6
10
−4
10
−2
10
0
10 Sampling time (s)
2
10
4
10
−5
0
10
20
30
40 50 Simulation time (s)
60
70
80
90
Fig. 4. Comparison of the simulated OMNeT++ C model and real-world oscillator behavior
Fig. 5. Clock offset of the virtual local clock compared to the ideal simulation time
the clock value) and rate (adapting the increment and thus the ”speed”), similar to typical precision clock implementation approaches in hardware, like the one described in [13]. According to the centralized architecture of the flex WARE system, the FC acts as a clock master, while all access points and mobile nodes are clock slaves. The clocks at the access points are implemented as synchronized transparent clocks. They are synchronized to the master like ordinary clocks, but act as transparent clocks for PTP packets passing to and from the node, in order to avoid cascaded control loops. The master clock at the controller periodically sends out synchronization messages. The slave clocks receive those messages and adapt their time and rate value, so that the clock rate best matches the measured clock rate of the master. This rate change has a severe impact on the simulation model of the local clock. As described, the local clock acts as interface between the local virtual time base and the simulation time. When the clock rate changes, the mapping of the local time to the global time changes. Therefore, all future events in the domain of the local clock would need to be rescheduled at the global event scheduler. To avoid this performance degradation, the local clock stores all future events in a local sorted queue and only schedules the next event. In case of a clock rate change due to a synchronization packet only one event per clock needs to be rescheduled, instead of all buffered future events.
time base, a simple simulation scenario based on the flex WARE framework was set up consisting of a flex WARE controller (FC), access point (FAP) and node (FN). To be compliant with the flex WARE system concept, the local clock of the FC was configured to be the master clock, and the FN as a slave. For the node synchronization, the FAP was configured as transparent clock in order to compensate for the latencies and jitter the sync packet experiences when traversing the FAP. The concept was validated in three steps:
V. R ESULTS The proposed method of a local, virtual time base for an event based simulation tool provides the possibility to simulate large-scale, distributed systems with a realistic simulation of the distributed clock domains. This is a key feature for evaluating and validating large-scale wireless systems with realtime capabilities. In the use-case of the flex WARE system, the realistic simulation model of distributed clocks is mandatory to test and verify the systems core concepts: In order to evaluate the proposed method of a local, virtual
A. Oscillator Model Evaluation Fig. 4 proves the validity of the used oscillator model as basis for the local clock module. Depicted is the measured Allan variance (y-axis) of a real-world oscillator for time intervals of increasing length (x-axis). The matching of the real-world behavior and the simulated one is shown. B. Evaluation of a Free Running Clock Fig. 5 gives an illustration of the difference between the requested time of an event in ideal simulation time and the actually scheduled event time, based on the calculation of the local clock model. For the sake of simplicity, the figure shows a periodically recurring event, with a period time of 1 s. The figure illustrates how much the simulation time, at which the event was actually scheduled, differs from the requested ideal period of 1 s for each occurrence of the event. C. Evaluation of the Clock Synchronization The results of the simulation of clock synchronization accuracy are shown in Fig. 6 and Fig. 7. The time offset of the two local clocks were calculated whenever a sync packet was received at the slave clock. The offset was calculated as the difference between the sending timestamp (recorded at the master) and the receive timestamp (recorded at the slave), compensated by the measured line delay, as defined in [12]. The sync packets were sent with a 0.5 s interval for a simulated duration of 500 s.
Number of observations
150
100
50
0 −5
−4
−3
Fig. 6.
−2
−1 0 Clock offset (s)
1
2
3
4 −8
x 10
Wired clock synchronization accuracy
Number of observations
80 60 40
R EFERENCES
20 0
is valid to accurately simulate high-precision clock synchronization in the range of nanoseconds by using an oscillator model. It is also valid for the simulation of synchronization systems which experience a high jitter on the transmission delay measurement, and it is valid to simulate systems which do not employ a clock synchronization mechanism. Results for clock synchronization simulation in the context of the wireless factory automation concept flex WARE were presented. Further steps will be to investigate and implement a more detailed model of the wireless channel transmission delay. The proposed simulation framework will also be used to conduct performance investigations of the overall flex WARE system for specific use-cases like automated guided vehicles or robotic applications.
3
3.5
4
Fig. 7.
4.5
5 5.5 Clock offset (s)
6
6.5
7
7.5 −5
x 10
Wireless clock synchronization accuracy
The high-precision clock synchronization on the wired connection between the controller and the access point (Fig. 6) was measured with a simulated hardware timestamper, where the jitter on the physical layer can be neglected. With this setup the achieved mean value was 9.6 ns with a standard deviation of 10.9 ns. As it is assumed that the flex WARE nodes will not have hardware timestamping units, the timestamper model in the FN’s case was configured to simulate jitter on all protocol layers to match the observed behavior of real systems, as presented in [4], with a white noise of 5 µs standard deviation. Fig. 7 shows the accuracy of the wireless clock synchronization for the same simulation run as in the above case. The calculated mean value was 55.4 µs and the standard deviation 5.7 µs. VI. C ONCLUSION AND O UTLOOK Discrete event simulators are the state-of-the-art tool to verify distributed communication systems. The reason is that the complexity of these kind of large-scale systems makes it unreasonable to perform verification by the usage of prototypes. For most communication protocols and applications the distributed clocks need to be synchronized. To test and verify such systems, clock synchronization needs to be simulated in a realistic manner. A problem of distributed clock synchronization simulation in discrete event simulators is the central event scheduler, which maintains only a single, global time base, which results in perfectly synchronized distributed clocks. This paper presented a solution to overcome this problem by introducing the local clock concept. It was shown that the local clock concept
[1] G. Gaderer, P. Loschmidt, and A. Mahmood, “A novel approach for flexible wireless automation in real-time environments,” in Proceedings of the 2008 IEEE International Workshop on Factory Communication Systems, G. Cena and F. Simonot-Lion, Eds., IEEE. IEEE, May 2008, pp. 81–84. [2] T. Sauter, The Industrial Information Technology Handbook, Part II (Industrial Information Technology), Section 3 (Industrial Communication Systems), Chapter 52. CRC Press LLC, 2000 N.W. Corporate Blvd., Boca Raton, Florida 33431: CRC Press, 2005, ch. Linking Factory Floor and the Internet, pp. 52–1 – 52–15, part II, Section 3, Chapter 52. [3] T. Sauter, J. Jasperneite, and L. Lo Bello, “Towards new hybrid networks for industrial automation,” in Proc. IEEE Conference on Emerging Technologies & Factory Automation ETFA 2009, 2009, pp. 1–8. [4] A. Mahmood, G. Gaderer, and P. Loschmidt, “Software support for clock synchronization over IEEE 802.11 wireless LAN with open source drivers,” 2010, to be published in Proceedings of the IEEE International Symposium on Precision Clock Synchronization for Measurement, Control and Communication ISPCS 2010. [5] G. Gaderer, P. Loschmidt, A. Nagy, R. Exel, and T. Sauter, “Localisation in Wireless Sensor Networks,” in Proceedings of the IEEE 2009 Sensors Conference, P. J. French, Ed., Oct. 2009, pp. 1004–1009, sensors 2009. [6] A. Varga, “Using the OMNeT++ Discrete Event Simulation System in Education,” IEEE Trans. Educ., vol. 42, p. 11, Nov. 1999. [7] X. Xian, W. Shi, and H. Huang, “Comparison of OMNET++ and Other Simulator for WSN Simulation,” in Proc. 3rd IEEE Conference on Industrial Electronics and Applications ICIEA 2008, Jun. 2008, pp. 1439–1443. [8] G. Gaderer, P. Loschmidt, and T. Sauter, “Quality Monitoring in Clock Synchronized Distributed Systems,” in Proceedings of the IEEE International Workshop on Factory Communication Systems, WFCS 2006, Jun. 2006, pp. 13–21. [9] J. Vig, “Introduction to Quartz Frequency Standards,” Army Research Laboratory, Electronics and Power Sources Directorate, Fort Monmouth, NJ 07703-5601, USA, Tech. Rep. SLCET-TR-92-1 (Rev. 1), Oct. 1992. [10] D. W. Allan, “Time and frequency (time-domain) characterization, estimation, and prediction of precision clocks and oscillators,” IEEE Transactions on Ultrasonics, Ferroelectrics and Frequency Control, pp. 667–674, 1987. [11] G. Gaderer, A. Nagy, P. Loschmidt, and N. Ker¨o, “A Novel, High Resolution Oscillator Model for DES Systems,” in Proceedings of the 2008 IEEE International Frequency Control Symposium, May 2008, pp. 178–183. [12] J. McKay, Ed., IEEE 1588 (tm) 2.1 Standard for a Precision Clock Synchronization Protocol for Networked Measurement and Control Systems. IEEE, March 2006. [13] P. Loschmidt, G. Gaderer, and N. Ker¨o, “IEEE 1588 Hardware for Fault Tolerance and High Precision,” in Proceedings of the 2006 International Workshop on IEEE 1588, Gaitersburg, Maryland, USA, Oct. 2006, pp. 1–6.
