Wireless Sensor Networks (WSN) have been, and still are, a rich field for scientific activity. The immense advantages of WSNs in terms of mobility and flexibility ...
Clock Synchronization Simulation for Wireless Sensor Networks Felix Ring, Anetta Nagy, Georg Gaderer, and Patrick Loschmidt Institute for Integrated Sensor Systems Austrian Academy of Sciences Wiener Neustadt, Austria Email: {felix.ring, aneta.nad, georg.gaderer, patrick.loschmidt}@oeaw.ac.at
Abstract—A lot of research is done in the field of wireless sensor networks, particularly in the field of ad-hoc networks. Large-scale real-time wireless networks for factory automation are hardly covered by existing research. Such networks usually have thousands of nodes, which makes simulation a necessity for system validation. Discrete Event Simulators (DES) are efficient tools to do so, but their event schedulers make it impossible to simulate distributed clocks, which is mandatory for validation of TDMA based protocols. This paper proposes an approach to enhance DES systems with models of local clocks. Further, it provides a realistic simulation of distributed clocks without losing the performance advantages of abstract modeling.
I. I NTRODUCTION Wireless Sensor Networks (WSN) have been, and still are, a rich field for scientific activity. The immense advantages of WSNs in terms of mobility and flexibility over traditional wired networks open a wide variety of possible applications. However, the advantages come at the price of new challenges like dynamically changing topologies, limited energy sources, limited memory, limited bandwidth etc. As shown in [1], the number of challenges together with the achievable benefits make WSN a rewarding field for research. Nevertheless, the topic of wireless factory automation has not received much attention during the past years. Wireless technology bears a lot of benefits for factory automation, as presented in [2]. Cabling costs can be reduced, processes can be dynamically adjusted and reconfigured, and location based services can introduce new possibilities in terms of safety and security, but also enable new levels of automation, e. g. for Automated Guided Vehicles (AGV). 1 The use-case of WSN for factory automation, as well as other typical use-cases, like environmental monitoring, health, surveillance, catastrophe monitoring, military, traffic monitoring often have real-time requirements for the communication system. Many of those applications are safety critical, and thus need to be thoroughly validated, verified, and tested. Typically a large number of nodes are involved. Due to hardware costs and production cycle times it is unreasonable to produce a high 1 The work presented in this paper is partly financed by the province of Lower Austria, the European Regional Development Fund, the FIT-IT project ε-WiFi under contract 813310, the EU under the FP7 STREP flex WARE contract number 224350, as well as by the Austrian Center of Competence in Mechatronics (ACCM) under contract A260501 and Oregano Systems Design and Consulting.
amount of prototypes for such systems to test new concepts. Discrete Event Simulators (DES) have proved to be a very efficient way to evaluate communication protocols and system concepts. The rise and fall of a successful simulation goes hand in hand with proper system models. Unfortunately, common DES are not capable of properly simulating distributed clocks. However, clock synchronization is a major topic for almost any type of WSN, especially when real-time protocols are involved. Therefore, this paper proposes a new concept for a realistic modeling and simulation of distributed clocks in DES. The presented work was inspired by requirements of the application field of factory automation, where sophisticated real-time protocols for infrastructure mode [3] as well as the localization scheme [4] require high-precision clock synchronization. However, the presented techniques for realistic simulation of distributed clocks in DES are by no means limited to this use-case. II. D ISTRIBUTED C LOCKS Many real-time communication protocols use a TDMA (Time Division Multiple Access) based approach in order to guarantee conflict free usage of a shared communication medium. Time, and especially timely delivery of messages, is a crucial system property. For a correct operation of such a communication system, all participants must have the same, or at least a very similar notion of time to be able to stick to the distributed time schedule. This is solved by synchronizing the clocks of the nodes. In wired real-time capable systems, basically two approaches exist to achieve clock synchronization. One approach is to directly connect the several participating modules with a dedicated clock line, so that all modules are controlled by a single master clock. It goes without saying that this approach is impossible to pursue in the wireless world. The second approach is to create clusters containing several nodes and synchronize the clocks within the cluster. A. Clock Synchronization in Wireless Sensor Networks Various techniques for clock synchronization in WSNs exist. A good overview of specific requirements and solutions for clock synchronization in WSNs is provided in [5]. In general, external and internal synchronization mechanisms
can be distinguished. For external synchronization, all clocks synchronize to the same (externally provided) time, while internal synchronization tracks different time offsets to other nodes. A survey of synchronization protocols of both kinds for suitability for WSNs is published in [6], [7]. B. Adder-Based Clocks In order to properly model clocks for clock synchronization simulation, the mechanisms of their real-world representatives need to be understood. In simple terms, a common clock can be viewed as a two-part device: It consists of an oscillating part (frequency standard) which determines the length of the second or a time interval (clock increment) and a counter, which keeps track of the number of oscillator ticks, hence the name adder-based clock. Due to many random and systematic trends, if two clocks are freely running, the difference in the times they are showing will increase more and more. The clock inaccuracies originate from the oscillator imperfections: The time error comes from the simple fact that the time increment is constant, although the frequency of the oscillator is not stable. C. Adder-Based Clocks in Clock Synchronization Clock synchronization schemes deal with setting two or more clocks to the same time, usually by using adder-based clocks with a variable increment. A hardware implementation of such a clock is described in [8]. By varying the value of the clock increment the “speed” of the clock can be controlled. As there is no such thing as absolute time, it is only possible to compare two clocks, where one is used as a reference and the second as the clock under test. Preferably, the reference should have a much more stable frequency standard. Clock synchronization protocols (e.g. the well-known IEEE 1588 [9]) often follow a master-slave approach. The principle is to equip the master clock with a much better frequency standard and synchronize the local clock of slaves to the master’s time. In the mentioned IEEE 1588 protocol the master clock’s time is periodically broadcasted in so-called sync packets to the slaves. The master’s time is compared to the local time and the clock increment is corrected in a control loop as shown in figure 1. The master-slave principle can easily be applied to WSNs when so-called anchor nodes are used. These nodes usually have a known location and can be equipped with more complex hardware, therefore it is recommended to use them as master clocks.
event from the list, awakes the issuing module to process the event and sets the simulation time-property to the time at which the event was specified to occur. The advantage of such a system is the high efficiency in terms of processing power, especially in the case of scarce events. However, one disadvantage is that there is no linear time-scale between events. Event synchronization by waiting (like the sleep() statement in C) is not possible. A further disadvantage is that inherently all components use the same, ideal clock. Therefore, we propose the use of local clocks to overcome this disadvantage. A. The Local Clock Concept A property of the centralized event scheduling system is a global time-scale. Whatever module requests an event at a certain time is awakened with the simulation time-property set to exactly this previously specified time. Any two modules, which schedule an event to occur at exactly the same time will be awakened with the simulation time-property being set to exactly the same time. This is true for all modules, even if they are located in nodes which represent distinct hardware nodes with their respective hardware clocks. This behavior would correspond to components with perfectly jitter-free and synchronized clocks. To evaluate the effect of imperfect clocks, as well as of the synchronization mechanism, a local clock needs to be introduced. Each component, which has its own clock in the real system to be simulated, needs its own instance of a local clock in simulation. Our proposed local clock acts as an interface between the event issuing modules of a component and the global event scheduler. In contrast to the continuous time-scale of the simulator, the local clock, together with the oscillator model, simulates a hardware clock with discrete ticks in time. Whenever an event is to be scheduled, the oscillator model calculates the tick closest to the requested time, considering jitter and drift parameters of the oscillator. B. Analytical Oscillator Model Since the oscillator is the part, which determines the behavior of the clock, the model of the clock will be as good as realistic the oscillator model is. It is well known that every oscillator has a certain inaccuracy due to deterministic (frequency offset, drift, etc.) and nondeterministic (like white noise, flicker noise, random walk) trends. The error in the
III. S IMULATION WITH D ISCRETE E VENT S IMULATORS Discrete Event Simulators (DES) like OMNeT++ [10], ns3 [11], or OPNET Modeler [12], are often used to simulate communication systems. They are very efficient, as they don’t simulate idle times, but are triggered only when an event, usually the transmission of a message, occurs. Each module requests an event at a given time in the future. The global event scheduler stores all pending events in a sorted list, and works through them, event for event. When one event has successfully been processed, the event scheduler takes the next
PID s time increment
ns
tslave tmaster
CLKevent Fig. 1.
Control loop with adder based clock
+
-9
2
x 10
1 0 Local time offset [ns]
oscillator frequency is random and uncorrelated with the errors in the past (the oscillator has no memory). However, as the adder-based clock is triggered by the oscillator ticks, it accumulates the time error, and thus is an implicit memory for the oscillator jitter. This behavior needs to be modeled properly. Often, the nondeterministic part of the oscillator noise is modeled with simple white noise or with Brownian motion, as described in this patent [13]. In that case the local time is calculated as:
-1 -2 -3
x(t) = x0 + y0 (t) + 1/2Dt2 + φ(t)
(1) -4
where t is the global time, x(t) is the local time, y0 and D are frequency offset and drift, respectively, and φ(t) models the deviation. A more profound strategy for oscillator modeling was presented earlier by the authors in [14], which takes the stochastic nature of oscillators into account. Proper measures of the oscillator instabilities are crucial for a realistic analytical model. As the signal to simulate is a random function of time, the Power Spectral Density (PSD) of the noise is a useful measure, as the PSD is time-invariant for the types of random noises which occur in oscillators. Once the PSD Sy (f ) is determined, it is possible to design a filter for the oscillator noise with the frequency response H(f ), which fulfills the equation: 2
Sy (f ) = |H(f )|
(2)
Once the filter is designed, the oscillator instabilities in the time domain are obtained by feeding samples of white noise into the filter for every simulated oscillator tick. Since it is not straight-forward to determine the PSD, the instabilities were first characterized by performing time-domain measurements. In the time domain the Allan variance [15] is used as a measure for the oscillator instabilities. The Allan variance is used rather than the standard deviation because it is convergent for all types of noises occurring in oscillators. Therefore, the oscillator modeling started by characterizing the instabilities using the Allan variance and finding a relation between the time and frequency domain. IV. C HALLENGES AND R ESULTING I MPLEMENTATION From the presented considerations it becomes clear that a system simulation with distributed clocks needs to implement the local clock concept. An advantage of most DES is the modular design. Therefore, we have implemented a general, reusable local clock module for the OMNeT++ simulation tool. The module consists of the following parts: • • •
Interface between the virtual time base of the local clock and the global simulation time of the event scheduler Oscillator model Adder-based clock with a variable increment
-5
0
10
Fig. 2.
20
30
40 50 60 Simulation time [s]
70
80
90
Local notion of time vs. global simulation time
A. Local Clock Time Interface The local clock module has to act as a time interface between the local representation of time with all the simulated imperfections and the global, perfect simulation time. The global simulation time is manifested in the central event scheduler of the DES. Therefore, the local clock needs to present an event scheduling interface between all modules controlled by the local clock and the central event scheduler. This is done by requiring all local modules (the ones which are controlled by an instance of a local clock) to register a callback function at initialization time at the local clock module. All events of a local module need to be requested at the local clock, instead of directly requesting it at the central simulation scheduler. The local clock then calculates, based on the oscillator model, the global simulation time which corresponds to its local notion of the requested time. When the time of the requested event has come, the local clock module gets activated by the central event scheduler and activates the requesting local module by calling its previously registered callback function. The difference between the global simulation time and the local notion of time at a local clock is displayed in figure 2. The graph shows the calculated local time for a module which requested a periodic event with a period of 1 s. B. Oscillator Model for DES In order to be able to schedule events, the local clock has to directly interact with the oscillator. Analytical models of oscillators could be applied in a straight-forward way by considering each oscillator tick as an event. However, this would contradict the very nature of discrete event simulation, require unreasonably high resources and would slow down the simulation significantly. Consequently, different approaches had to be considered. As the local clock’t time is proportional to the oscillator ticks, the oscillator needs to be capable provide the global simulation time to the local clock at a requested number of
Time at current event
Time at future event
Fig. 3.
t
-10
1) Get current ticks 2) Time at next tick
Measured Modeled
-8
10
10
F (t) > 0 N
N+1
F (t) < 0 N
N+1
Error caused by the simple oscillator model
Allan variance
Global time
-6
10
-12
10
-14
10
-16
10
-18
10
-20
10
oscillator ticks and to provide the number of ticks at a certain global time. In addition, the oscillator has to be able to draw samples for points in time that are close to each other without causing inconsistent behavior. For example, the N + 1th tick always needs to occur after the N th tick, and the global time ti should always be represented by the same number of elapsed oscillator ticks. In this work the analytical model based on the Allan variance was taken as a basis for the DES implementation. While simple oscillator models, which are using only white noise are acceptable for simulation tools with a continuous time base (e. g., SPICE), they can easily cause problems in discrete event simulators. A typical example is the calculation of the new time increment upon reception of a synchronization packet. This needs to be done in two steps: 1) The current number of oscillator ticks (N ) needs to be determined for the current global time 2) The new clock increment is calculated at the next tick (N + 1) Consider a situation when the current global time is very close to the N + 1th rising edge (see figure 3). Assuming that in the first step the value of the noise φ(t) is a positive number, the returned oscillator tick for the current global time is N . However, if in the second step the added random noise is negative, the returned global time can be lower than the current global time. Hence, the clock will attempt to schedule an event to the past, which is a logical contradiction. The proper modeling of the oscillator noise based on the Allan variance makes it possible to correctly schedule events respecting the timely order of the oscillator ticks. Since the oscillator noise at the N th thick is obtained by filtering N number of samples of white noise, the clock error at the current time/tick is dependent on the previous values. However, it has to be noted that this approach would be quite ineffective if the global time would be calculated from the oscillator start-up time for every request. One solution is to develop a caching strategy by storing the last calculated simulation time and number of ticks (together with the current filter values) whenever an event is processed. These values are then used later on as reference points for calculating future events. Figure 4 shows a comparison of the measured Allan variance
-6
10
Fig. 4.
-4
10
-2
0
10 10 Sampling time [s]
2
10
4
10
The Allan variance of the real and modeled oscillator
of a real world oscillator and of the simulated oscillator. It can be seen that the model shows the typical long term behavior of real oscillators. C. Clock Model Considerations with Clock Synchronization The principle of rate controlled clock synchronization with adder-based clocks is to change the value of the time increment of a clock in order to steer the clock’s time in the desired direction. For example, in IEEE 1588 upon reception of a sync packet the synchronization engine recalculates the value of the clock increment. From that point on, the value of the adder-based clock will be increased by the newly calculated increment at every oscillator tick. With the assumption that an event is scheduled at a certain local time, it is clear that by changing the time increment the event will occur at a different oscillator tick than before the increment was changed. In hardware this is no issue at all, however, due to the abstracted representation of time in simulation it requires special handling. The interaction with the oscillator model can be done in the following steps: 1) The clock needs to calculate the new number of ticks at the local time using the new increment 2) The oscillator model has to determine the new global time at the calculated ticks 3) The clock needs to reschedule the event to the new global time. The principle of rescheduling is graphically shown in figure 5. Due to the fact that the clock module can provide timer interfaces for multiple modules inside one sensor node, the number of the future events can be relatively high. In order to avoid the rescheduling of every event registered at the clock, the approach is to keep a local list of the events inside the clock module. In this local list the events are sorted according to the local clock time and only the next event is scheduled at
Event
Local time Ticks Global time
t
tevent = Tbefore
tevent = Tafter
t
Before rescheduling
t
After rescheduling
Fig. 5. Rescheduling local clock events due to change in the time increment
the central event scheduler. Naturally, in case there is a new timer request, which ought to occur before the current first event, the current first event has to be canceled and the new request needs to be scheduled. V. C ONCLUSION AND O UTLOOK The work presented in this paper enables system evaluation of large wireless sensor networks by simulation with regard to a realistic model of distributed clocks. Discrete event simulators are powerful tools for system evaluation of large systems. However, out-of-the-box simulation systems do not support realistic models for distributed clock simulation. This paper showed how a single clock can be realistically modeled by the use of a noise shaping oscillator model. It further showed how a system with distributed clocks can be simulated in DES by introducing the local clock concept, which allows a heterogeneous notion of time at each local clock. Further considerations about the interaction of the local clock concept with clock synchronization algorithms were discussed, and solutions were shown how to overcome challenges arising from the simulation in a discrete event simulation system. Further improvements can be made with regards to the required computational power of the oscillator model. Additionally, the distributed clock simulation concept can be combined with energy consumption simulation frameworks. By doing this, a powerful tool for the evaluation of various clock synchronization schemes, real-time algorithms, and system architectures would be at hand, providing useful information about the timely accuracy and power efficiency of the systems under test. ACKNOWLEDGMENT The authors would like to thank Martin Schimandl from Oregano Systems for the fruitful discussions, his fresh ideas and contribution to the presented work. R EFERENCES [1] I. F. Akyildiz, W. Su, Y. Sankarasubramaniam, and E. Cayirci, “Wireless sensor networks: a survey,” Computer Networks, vol. 38, no. 4, pp. 393– 422, Mar. 2002.
[2] G. Gaderer, P. Loschmidt, and A. Mahmood, “A novel Approach for Flexible Wireless Automation in Real-Time Environments,” in Proc. of the IEEE International Workshop on Factory Communication Systems, WFCS, G. Cena and F. Simonot-Lion, Eds., IEEE. IEEE, May 2008, pp. 81–84. [3] L. LoBello and E. Toscano, “An Adaptive Approach to Topology Management in Large and Dense Real-Time Wireless Sensor Networks,” IEEE Trans. Ind. Informat., vol. 5, no. 3, pp. 314–324, Aug. 2009. [4] G. Gaderer, P. Loschmidt, A. Nagy, R. Exel, and T. Sauter, “Localisation in Wireless Sensor Networks,” in Proc. of the IEEE Sensors Conference, P. J. French, Ed., Oct. 2009, pp. 1004–1009. [5] J. Elson and K. R¨omer, “Wireless sensor networks: a new regime for time synchronization,” SIGCOMM Comput. Commun. Rev., vol. 33, no. 1, pp. 149–154, 2003. [6] B. van Voorst. Time synchronization in wireless sensor networks. [Online]. Available: http://citeseerx.ist.psu.edu/viewdoc/download?doi= 10.1.1.80.6280&rep=rep1&type=pdf [7] H. Abubakari and S. Sastry, “IEEE 1588 Style Synchronization over Wireless Link,” in IEEE International Symposium on Precision Clock Synchronization for Measurement, Control and Communication, Sep. 2008, pp. 127–130. [8] P. Loschmidt, G. Gaderer, and N. Ker¨o, “IEEE 1588 Hardware for Fault Tolerance and High Precision,” in Proc. of the 2006 International Workshop on IEEE 1588, Gaitersburg, Maryland, USA, Oct. 2006, pp. 1–6. [9] “IEEE Standard for a Precision Clock Synchronization Protocol for Networked Measurement and Control Systems,” IEEE Std 1588-2008 (Revision of IEEE Std 1588-2002), pp. c1–269, Jul. 2008. [10] A. Varga and R. Hornig, “An overview of the OMNeT++ simulation environment,” in Simutools ’08: Proceedings of the 1st international conference on Simulation tools and techniques for communications, networks and systems & workshops. ICST, Brussels, Belgium, Belgium: ICST (Institute for Computer Sciences, Social-Informatics and Telecommunications Engineering), 2008, pp. 1–10. [11] (2007, October) ns-3 software architecture. [Online]. Available: http://www.nsnam.org/docs/design.pdf [12] X. Chang, “Network simulations with opnet,” in WSC ’99: Proceedings of the 31st conference on Winter simulation. New York, NY, USA: ACM, 1999, pp. 307–314. [13] H. Zhu, L. Ma, and B. K. Ryu, “System and method for clock modeling in discrete-event simulation,” U.S. Patent US 2009/0 119 086 A1, May 7, 2009. [14] G. Gaderer, A. Nagy, P. Loschmidt, and N. Ker¨o, “A Novel, High Resolution Oscillator Model for DES Systems,” in Proc. of the 2008 IEEE International Frequency Control Symposium, May 2008, pp. 178– 183. [15] D. W. Allan, N. Ashby, and C. C. Hodge, The Science of Timekeeping. Hewlett Packard, 1997, pp. 60–64.
