Nov 28, 2012 - IBM R-50 laptops, equipped with IEEE 802.11b compatible wireless cards ..... partners: Telkom SA Ltd, Huawei Technologies SA (Pty). Ltd and ...
Comparison of Routing Protocol Performance Using Wireless Mesh Network Simulation and Testbed Olukayode A. Oki, Pragasen Mudali, John B. Oladosu, Matthew O. Adigun and Bethel Mutanga Department of Computer Science University of Zululand, Private Bag X1001, KwaDlangezwa 3886 Tel: +27 35 9026012, Fax: +27 35 9026569 Email: {okikayode, johnoladosu}@gmail.com; {mudalip, adigunm}@unizulu.ac.za
Abstract- Wireless Mesh Networks (WMN) have received considerable attention as a means to achieve connectivity in community and commercial networks. Protocols at the different stack layers have varying effects both on the energy consumption and the general performance of WMN backbone nodes. Previous studies have shown that, simulations have often been used to obtain protocol performance data, but simulation results are at best an abstraction of the real world network performance. Thus, simulations often provide an unrealistic measure of protocol performance and testbed-based studies are proving more realistic. This paper analyses the performance of AODV and OLSR routing protocols in WMN using both a Simulator and an indoor testbed. Although, the simulation and testbed results aligned with each other for Packet delivery ratio and average throughput, but the magnitude of the difference between the two results is wide. However, the simulation result contradicts that of the testbed for the lifetime. Based on the results of this study, we argue that simulation results only give a rough estimate of the real world network performance. Hence, whenever it is feasible, validating simulation results using a testbed is highly recommended in order to improve our understanding of protocol performance. Index Terms— AODV, Lifetime, OLSR, Simulation, Testbed, Wireless mesh network I. INTRODUCTION Wireless Mesh Network (WMN) is still at a pre-industrial stage. However, the continuous demand for a cost-effective broadband connectivity, both in the rural and urban areas is paving way for the high increase in the number of WMN deployment. The deployment, maintenance and troubleshooting of WMN are facing some challenges, which have not been met. These challenges include: manual configuration of access nodes, lack of reliable power supplies to the nodes most especially in the rural areas of the developing nations [1] and the physical placement of the nodes in different terrain [2]. Due to the complexities involved in setting up a testbed, simulations are widely used by researchers to test and validate protocol performance in WMNs. The simulation environment presents a high degree of control, flexibility and produce results that are repeatable. In addition, simulation is very useful, when studying a highly distributed wireless mesh network. Simulation tools are very flexible, scalable and cost-effective, because with only one
suitably configured computer, complex experiments can be undertaken [3]. However, due to some unpredictable complications and dynamic behaviour of WMN in real environment, which simulator developers were not effectively captured; has led to some inconsistencies in the simulation results presented by the researchers. Hence, this has led to the tendency towards researchers complementing and validating simulation results with a testbed. Testbed and Simulation are two important evaluation methods, because they complement each other. However, testbed evaluations are normally conducted by implementing a prototype; hence, the results and conclusions can be easily transferred to real life, since the prototype and the results obtained from a testbed possess a higher-degree of reality than its simulation-based counterpart. Previous studies [4, 5, 6] have pointed out that, the inconsistencies in the results obtained from the same protocol running on different simulation tools is majorly due to the limitations introduced into the modelling of radio propagation by different simulator developers. However, it is not clear how the imperfect modelling affects the protocols performance and to what degree. It is therefore the goal of this study to bridge this gap. This paper presents a performance analysis of Wireless Mesh Network using simulation and battery-powered indoor testbed. Two common routing protocols proposed by IETF MANETs group [7] for adhoc networks were considered. The general performance and energy-efficiency of both Adhoc On-demand Distance Vector (AODV) and Optimized Link State Routing (OLSR) on simulation and indoor testbed were presented. AODV and OLSR represent reactive and proactive routing protocols respectively. Payload sizes and Transmission Power levels were varied to determine their impacts on the performance and energy-efficiency of the evaluated Routing protocols both in a simulation and in a testbed environment. In addition to Packet Delivery Ratio and Throughput, energy-efficiency was measured using the Node Lifetime. The overall results reported in this paper for both simulation and the testbed experiments shows that, OLSR at maximum payload size and maximum transmission power level outperform other scenarios in the achieved performance for packet delivery ratio and average throughput. While the AODV at minimum payload size and maximum transmission power level outperform other simulation-based scenarios in terms of the lifetime. And for the testbed-based scenarios, AODV at maximum payload size and minimum transmission power level outperform other scenarios, with regards to the lifetime.
The remainder of this paper is organised as follows. Section II presents a review of existing studies in the comparison of simulation and testbed performance in WMNs. Section III details the testbed and simulation setup employed while Section IV discusses the measurement methodologies used in this study. Section V discusses the performance evaluation results for both simulation and testbed, whilst the paper is concluded in Section VI. II. LITERATURE REVIEW Due to the cost, complexity and difficulties involved in implementing a testbed, simulations are widely used by researchers to test and to validate protocol performance in a wireless mesh network. However, there have been wide variations in the results presented by different researchers for the same protocols when different simulators are used. Hence, there is a need to validate existing simulation-based results using a testbed. Previous studies have investigated and analysed the wireless mesh network behaviour on a testbed. However, few studies have compared their testbed results with that of a simulator. In [4], an experimental analysis of AODV and OLSR over IEEE 802.11b/g in a static multi hop wireless adhoc network was presented. An indoor testbed was used to test the validity of some of the previous simulation results that have been presented in the literature. A static chain topology along with different hop counts was used. This indoor testbed comprises of five nodes, which were made up of IBM R-50 laptops, equipped with IEEE 802.11b compatible wireless cards and running Linux kernel 2.6.12 with ipw2200 driver. The transmission power of the wireless cards was set to the minimum value allowed by the manufacturer (-12dbm) in order to reduce the transmission range and to force a multi hop network. Four different scenarios were considered with various hop counts that range from 1 to 4. Whilst two performance metrics were measured: throughput and retransmission index (“the percentage of segments re-transmitted by the sender TCP”). The results of this study were presented alongside with simulation results. Some of these experimental results are in contrast with that obtained via simulation and such discrepancies were largely due to the different protocol implementations; where the one in real practice differs from that of simulation tools. Another reason for the discrepancies in the results was the existence of several wireless access points within the vicinity which led to some interference in the transmission, which was not captured in the simulation model. In [8], an experimental performance analysis of simulation tools and testbed tools in a wireless mesh network was presented. The main objective of the study was to investigate the discrepancies between the simulation tools and the testbed tools. The study focused on Physical, MAC and IP layer for both simulation and testbed. The factors considered, in investigating the discrepancies include: path loss, routing stability, transmission rate and interference. NS2 and QualNet were used as the simulator, while the testbed comprises of nodes which were made up of Hp nc6000 laptops, equipped with HP W500 802.11a/b/g wireless cards and running Linux kernel 2.6.25 with MadWiFi driver. The simulation results were presented alongside with testbed results. Some of the simulation results
are in contrast with that of the testbed and such discrepancies were largely due to the inadequate interference and inaccurate channel modelling in simulators. In [9], an experimental validation of static wireless mesh network performance was carried out using simulation, emulation and a testbed. For the sake of homogeneous comparison, the NS2 radio propagation model was calibrated to be similar to that of the testbed and the same routing protocol implementation was used for both the simulation, emulation and the testbed. The main objective of the study was to investigate the difference between the results obtained via the various evaluation methods. In order to achieve the objective, only the OLSR routing protocol was considered; using the following factors (packet latencies, packet delivery ratio and network topology) for validation. NS2 was used as the simulator, while the testbed comprise of 16 nodes, which were made up of Dell D410 laptops, equipped with 3Com PCMCIA wireless cards and running Linux with madwifi-ng driver. The WMN emulation was implemented, using NS2 emulation module and their own extensions for wireless emulation. The simulation results were presented alongside with emulation and testbed results. Based on the results of this study, it was concluded that the network topology and packet delivery ratio can be accurately represented in NS2 simulator, provided the simulation parameters are correctly modified. However, there was a contrast between the simulation packet latencies results and that of the testbed results, hence, future development of simulators were recommended. In all the previous studies discussed, laptops were used as the nodes for the testbed. Laptops are not ideal platforms for the study of the energy-efficiency and general performance of Routing Protocols as they contain significant processing and memory resources which would affect the throughput and the energy consumption of the network nodes. In addition, laptops are costly, multi-purpose devices. Thus, these devices are not typically employed in WMN deployments as backbone devices. Also, most of the testbed and simulation setup are not homogeneous, hence, this would affect the results presented. The performance analysis of routing protocols using simulation and testbed has been widely studied but the testbed evaluation platforms and general configurations do not closely resemble an actual WMN deployment. The analysis presented in this paper differs from previous studies in the following respects. In contrast to the use of laptops, this study employed battery-powered Linksys WRT54GL routers, which are popular WMN nodes. Both minimum and maximum transceiver power levels and packet sizes were used, whereas power levels were not previously considered in the simulation versus testbed evaluation. Average throughput and packet delivery ratio were analysed and the energy efficiency of AODV and OLSR is evaluated using the Lifetime metric. III. TESTBED AND SIMULATION SETUP The study utilized both a simulator and an indoor WMN testbed, to compare the general performance of AODV and OLSR protocols. The testbed comprises of fourteen nodes, which were arbitrarily placed in an 8m x 12m room. The availability of plug points influenced the node placement, which is analogous to the coupling of nodes with existing
infrastructure such as street lights and buildings in realworld deployments. The nodes were labeled from N1 to N14 for the purpose of clarity, as depicted in Figure 1. All the nodes, apart from Node N1 were mains-powered. Node N1 was powered using a 12V8Ah battery connected to a Digital Multi-Meter (DMM). The DMM was connected to a data collection PC, via an Ethernet link [See Figure 1]. This testbed is located within a faculty building with labs and offices. Hence, there are several other wireless LANs that are operational within the proximity. The testbed was operated in 802.11g mode at 2.4GHz on channel 6 so as to mitigate against the interference caused by other wireless LANs operational within the building. Cisco Linksys WRT54GL v1.1 routers with OpenWRT Freifunk v1.7.4 firmware were installed to provide mesh functionality and each of these routers used as the nodes, are equipped with a pair of 5dBi gain antennae. These antennae were disconnected in order to reduce the transmission range so as to be able to force a multi-hop wireless network within an indoor environment. The Linksys WRT54GL router was chosen for this study, because it is a popular choice for WMN deployments such as in [10-12], due to its low cost and easy availability. The WRT54GL routers possess a 200MHz processor, a Broadcom 802.11b/g radio chipset, 4MB flash memory and 16MB of RAM. The wireless chipset allows transmission power output levels to be set between 1 and 19dBm, which is the maximum power output recommended by the manufacturer. The two most widely used Transport Layer protocols are TCP and UDP. However, this study uses Transmission Control Protocol (TCP) as the Transport Layer protocol because of the UDP performance in our previous study [1]. NetScanTools Pro version 11.0 was utilized to generate the TCP traffic. The NetScanTool application default settings were used for this study, except for the packet data length (either 32 or 512 bytes) and the number of packets sent (dependent upon the node lifetime achieved). The transmission power levels used by the testbed nodes were varied by using either the minimum or maximum level (1dBm or 19dBm). The wl utility was employed to adjust the transceiver power levels while the Digital Multi-meter (DMM) data capture was used to capture real-time node operational lifetime voltage. The Network Simulator version 2.34 (NS2) software running on Ubuntu 10.04 Operating System was used to conduct an extensive simulation for this study. NS2 is an open-source event-driven simulator tool that was designed particularly for research in computer communication networks [13]. The wireless nodes in this simulation study were modelled on a Cisco Linksys WRT54GL v1.1 router [14] using NS2.34. This particular router model was used for our simulation in order to have an homogeneous setup as that of the testbed, so as to be able to compare the testbed results with that of simulation. All the protocols (Routing and Transport) used were the ones implemented according to the corresponding RFC standard. The “setdest” utility of ns-2.34 was used to generate Scenario files, while the “cbrgen.tcl” utility was also used to generate the TCP traffic files. tTcl scripts were executed in order to generate the trace files for various protocols (OLSR, AODV & TCP). The detailed trace files
Table 1: SIMULATION SETUP DETAILS
Simulation Time
1000 Seconds
Number of Nodes
14 nodes
Network Area
600m x 600m
Channel/Frequency
6/2.34 GHz
Mac protocol
IEEE 802.11
Transport Layer Protocol
TCP
Traffic type
CBR
Packet Size
32 bytes, 512 bytes
Rate
4 kb/s
Nodes movement
Static
Initial Energy
3.0 Joule
Transmit Energy
1.6W
Receive Energy
1.3W
Figure 1: The architecture of the Mesh testbed generated from the various simulation experiments were stored and analyzed using an AWK script, while Microsoft Excel and Gnuplot were used to plot the graphs. Table 1 summarizes the additional simulation setup details that were used in this study. IV. MEASUREMENT METHODOLOGY In this study, three performance metrics were considered, and these include: Packet Delivery ratio, average throughput and Node lifetime. However, the Node lifetime is the main metric, studied with the objective of measuring the energy efficiency of both AODV and OLSR routing protocols. In achieving more statistical accuracy, each of the testbed and
simulation experiments was repeated five times. Each of the testbed experiments lasted a minimum of 30 hours and all evaluation data was collected at PC1 via an Ethernet link to Node N1 and a USB connection to the DMM (see Figure 1). The data collected via Ethernet and USB had no effect on the communications via the wireless interface of the testbed node. The use of a combination of Routing protocols, packet sizes and transmission power levels resulted in the eight evaluation scenarios recorded in this study (see Table 2). The following measurement procedure was used for each of the metrics measured. 1) Node Lifetime: The node lifetime was determined by measuring the elapsed time until the battery discharges from a fully-charged 12.5V to a pre-defined threshold voltage of 10.5V. The threshold voltage value was derived based on initial experimentation with the battery. This value is considered safe in order to ensure that the battery’s capacity is not impaired during the discharge process. Data packets were sent until the threshold voltage was reached. While for the simulation, the lifetime was defined as the length of time until the first node exhausts its energy allocation. The number of surviving nodes is determined and plotted against the simulation time.
Figure 2a: Simulation-based Packet Delivery Ratio at Maximum Transmission Power
2) Throughput: This is determined by the number of data packets that were processed over a period of time. The throughput is measured at the application layer and the data traffic is generated between the source node and the destination node, using NetScanTool 11.0 application for the testbed and cbrgen.tcl utility for simulation. Both the number and size of the data packets could be varied. Payload size of 512 and 32 bytes were employed in this study and the higher the throughput value, the better the network performance. 3) Packet Delivery Ratio (PDR): PDR is defined as the fraction of all the data packets from the sender node that reaches the destination node at the application layer. Different payload sizes (32 and 512 bytes) were specified using NetScanTool 11.0 application and “cbrgen.tcl” utility. An optimal route needs to have high PDR, hence, the higher the PDR value the better the network performance.
Figure 2b: Testbed-based Packet Delivery Ratio at Maximum Transmission Power Table 2: Evaluation Scenarios AODV at maximum transmission power with minimum payload AODV at minimum transmission power with minimum payload
V. EVALUATION RESULTS The results of the performance evaluation of both AODV and OLSR in simulation and battery-powered indoor testbed are presented here. Each of the reported graphs is the average of the results of five experiments for each setup. A. Packet Delivery Ratio Packet Delivery Ratio measures the effectiveness of Routing protocols in delivering traffic to the intended destinations. The performances of both AODV and OLSR when subjected to differing packet sizes and transmission power levels using both simulation and testbed are shown in Figures 2a & 2b respectively. In Figures 2a and 2b, it can be observed that OLSR at maximum transmission power level with maximum payload size outperform others, while AODV at minimum transmission power level with minimum payload size for the testbed was the least performing scenario in terms of PDR. It can also be observed that AODV performs better in the
AODV at maximum transmission power with maximum payload AODV at minimum transmission power with maximum payload OLSR at maximum transmission power with minimum payload OLSR at minimum transmission power with minimum payload OLSR at maximum transmission power with maximum payload OLSR at minimum transmission power with maximum payload
simulation scenarios, but performed poor on the testbed. One of the major reasons for the AODV poor performance in testbed is high route failure during the packet transmission. The route failures on the testbed are mostly caused by interference from other WLANs within the proximity. Some other previous testbed studies [15, 16] had
also reported that AODV experienced more than 80% route failure on the testbed for both maximum and minimum payload sizes. However, this is not the case with the simulation results, because interference is not effectively captured in the simulators. Hence, it makes the simulation results not accurate enough, since interference is unavoidable in real life deployment. B. Throughput Throughput measures the rate at which data is being received by the intended destination. Figures 3a and 3b depict the throughput levels of the AODV and OLSR when subjected to differing packet sizes and transmission power levels using both simulation and testbed respectively. The simulation throughput was measured in kilo bits per seconds (kbps), while that of testbed was measured in bits per seconds (bps). It can be observed from these results that OLSR at maximum payload size and maximum transmission power levels outperformed other scenarios in both simulation- and testbed-based. While the testbed AODV at minimum payload size and minimum transmission power is the least performing scenario. However, both OLSR and AODV at minimum payload size for both simulation and testbed performed very poor. Although, the simulation results aligned with that of testbed results, but the magnitude of the difference between the two results are wide. One of the reasons that can be attributed to the wide magnitude between the two results is the way the physical layer model of the simulators is designed. NS2 and most other common simulators have a two-ray path loss model and their signal reception is based on signal-to-noise ratio threshold; all of which, significantly contribute to the simulation results. C. Node Lifetime Figures 4a and 4b depict the effect of AODV and OLSR on the operational lifetime of a battery-powered WMN, when subjected to various payload sizes and transmission power levels, using both simulation and testbed. It can be observed from Figure 4a that AODV at minimum payload and maximum transmission power level outperforms other simulation scenarios in terms of lifetime, as it powers the network for approximately 800seconds simulation time.
Figure 3b: Testbed-based Throughput at Maximum Transmission Power
Figure 4a: Simulation-based Network Lifetime
Figure 3a: Simulation-based Throughput at Maximum Transmission Power Figure 4b: Testbed-based Network Lifetime
While for Figure 4b, AODV at maximum payload size and minimum transmission power level outperform other testbed scenarios in terms of node lifetime, as it powers the node for approximately 136,205seconds. It can be observed from Figures 4a & 4b, that the simulation-based lifetime result is in contrast to that of the testbed-based result. These discrepancies between the simulation and the testbed results can be attributed to different assumptions in the protocols implementations used in real practice and that of the simulation tool. Another reason is the variation in the available hardware and software parameter settings used in real practice and that of the assumed ones (like interference free, stable link/route establishment), used in simulation tools. VI. CONCLUSION The general performance and energy consumption behaviour of both AODV and OLSR were analysed, using both the simulation and an indoor battery-powered WMN testbed. We analysed how different transmission power levels and payload sizes affect both the energy usage and the general performance of AODV and OLSR in WMNs. The overall results of both the simulation and the testbed experiments show that, OLSR at maximum payload size and maximum transmission power level outperformed other scenarios in the achieved performance for packet delivery ratio and average throughput. While the AODV at minimum payload size and maximum transmission power level outperform other simulation-based scenarios in terms of the lifetime. And for the testbed-based scenarios, AODV at maximum payload size and minimum transmission power level outperformed other scenarios in terms of lifetime. Based on the results of this study, we argue that simulation results only give a rough estimate of the real world network performance. Hence, whenever it is feasible, validating a simulation results using testbed is highly recommended in order to have clear and better understanding of the protocol performances. Besides serving as a means to test and refine the practical applicability of AODV and OLSR routing protocols in WMNs, the testbed also allowed us to study many practical issues that inspire future directions on how to refine the AODV source code so as to address the route/link failure problem. We also intend to repeat the experiments on an outdoor testbed being planned on the University campus. ACKNOWLEDGMENT This work is based on the research supported in part by the National Research Foundation of South Africa-Grant UID: TP11062500001 (2012-2014). The authors also acknowledge funds received from industry partners: Telkom SA Ltd, Huawei Technologies SA (Pty) Ltd and Dynatech Information Systems, South Africa in support of this research. REFERENCES [1] Oki, O.A. Mudali, P. Mutanga, B. & Adigun M.O, “Evaluating the energy-efficiency of transport layer protocols in battery-powered WMNs”, In Proc. IET Intl Conf. on Wireless Communications & Applications, (ICWCA) Malaysia, 2012.
[2] Robinson, J. Singh, M. Swaminathan, R. Knightly, E. “Deploying Mesh Nodes under Non-Uniform Propagation”, In Proc. of IEEE INFOCOM’10, 2010. [3] Zimmerman, A., Gunes, M., Wenig, M., Meis, U., & Ritzerfeld, J. “How to Study Wireless Mesh Networks: A hybrid Testbed Approach”, In Proceedings of the International Conference on Advanced Networking and Applications (AINA) pp. 853-860, 2007. [4] Anastasi, G. Ancillotti, E. Conti, M. & Passarella, A. “Experimental analysis of a transport protocol for ad hoc networks”. In Proceedings of the ACM International workshop on Performance Evaluation of Wireless Ad hoc, Sensor and Ubiquitous Networks, pp. 9-16, 2006. [5] Gregori, E. Conti, M. Borgia, E. & Anastasi, G. “Wi-Fi in Ad Hoc Mode: A Measurement Study”. Proc. of IEEE International Conf. on Pervasive Computing & Communications, PerCom’04, Florida, 2004. [6] Johnson, D. “Performance Analysis of Mesh Networks in Indoor and Outdoor Wireless Testbeds”. University of Pretoria, South Africa 2007. [7] “IETF Mobile Ad-Hoc Networks (MANET) Working Group,” 28 November 2012, http://www.ietf.org/html.charters/manet-charter.html. [8] Tan, K. Wu, D. Chan, AJ. Mohapatra, P. “Comparing Simulation Tools and Experimental Testbeds for Wireless Mesh Networks,” Pervasive and Mobile Computing Journal, pp. 434-448, 2011. [9] Ivanov, S. Herms, A. Lukas, G. “Experimental validation of the ns-2 wireless model using simulation, emulation, and real network”, In: Proceedings of the 4th Workshop on Mobile Ad-Hoc Networks, 2007. [10] Amir, Y. Danilov, C. Hilsdale, M. Musǎloiu-Elefteri, R. & Rivera, N. “Fast handoff for seamless wireless mesh networks”. In Proceedings of the 4th international conference on Mobile Systems, applications and services (MobiSys '06) pp. 83-95, 2006. [11] Ismael, J. Bury, S. Pezaros, D. & Race, N. “Deploying Rural Community Wireless Mesh Networks”, IEEE Internet Computing, 12(4), pp. 22-29, 2008. [12] Lundgren, H. Ramachandran, K. Belding-Royer, E. Almeroth, K. Benny, M. Hewatt, A. Touma, A. & Jardosh, A. “Experiences from the Design, Deployment, and usage of the UCSB MeshNet Testbed,” IEEE Wireless Communications, 13(2), pp. 18-29, 2006. [13] Fall, K. & Varadhan, K. “The ns Manual –the VINT Project” www.isi.edu/nsnam/ns/ns-documentation.html 2008. [14] Linksys Inc. http://www.linksys.com 2011. [15] Brolin, J. & Hedegren, M. “Packet Aggregation in Linux”. Karlstads University, 2008 [16] Johnson, D.L. & Lysko, A. “Comparison of MANET routing protocols using a scaled Indoor wireless grid”. Mobile Networks & Applications, 13(1), 2008.
