Oct 19, 2005 - Douglas C. Sicker, Tom Lookabaugh, Jose Santos, Frank Barnes. Interdisciplinary Telecommunications Program. University of Colorado at ...
Assessing the Effectiveness of Remote Networking Laboratories Douglas C. Sicker, Tom Lookabaugh, Jose Santos, Frank Barnes Interdisciplinary Telecommunications Program University of Colorado at Boulder Boulder, Colorado, 80309 [douglas.sicker, tom.lookabaugh, jose.santos, frank.barnes]@colorado.edu Abstract - The Interdisciplinary Telecommunications Program at the University of Colorado has developed an internet based remote laboratory environment for master’s level graduate students; our suite of telecommunications experiments substantially extends prior work focused on networking equipment, by (1) providing a systems focus, (2) enabling multiple reinforcing methods of accessing the educational material, (3) providing a configuration matrix to support realtime network reconfigurations (of real network elements) (4) undertaking a careful assessment of the learning environment. The goal was to create an environment that reproduced (not just emulated) the lab experience. We recently completed the final phase of this project focusing on assessment of this learning tool; such assessment is still rare in the literature on remote laboratories. We describe the project from three perspectives; students’ exam results, students’ lab reports, and students’ satisfaction with the distance experience (based on interviews). We conclude that our remote laboratories provide similar learning outcomes to their in class analogues, but that there are important differences in student perceptions of the experience, including perceived difficulty and pace. Index Terms – Remote laboratory, education, telecommunications. INTRODUCTION Laboratory experience reinforces learning of theoretical concepts and provides the translation from theory into practical understanding; however, the cost, availability and physical access to laboratories have limited the number of students who can receive such education. The problem may be worsening according to a 2000 American Society for Engineering Education workshop report calling for universities to “find innovative ways for satisfying objectives” impacted by declining financial support for laboratories . One such innovative approach relies on the Internet, which provides an almost universally available communications infrastructure for learning engineering theory. The use of the Internet to also support the laboratory experience has been an area of increasing interest and experimentation [1-18]. Over the last three years, the Interdisciplinary Telecommunications Program (ITP) at the University of Colorado has developed its Remote Laboratory Infrastructure
(ReLI at http://ReLI.colorado.edu) environment to provide master’s level graduate students with access to laboratory resources through the Internet. The initial work was a proof of concept for the development and use of virtual instruments, simulated experimental environments, and the remote control of physical equipment by students to enhance the quality of their learning in telecommunications. Our suite of networking experiments substantially extends prior work by: • • • •
Providing remote operation of real networking equipment Focusing on systems level learning Providing remotely reconfigurable environments Assessing the educational outcome
The goal is to create an environment that reproduces (not just emulates) the lab experience. Initially, we focused on assessing the student needs and developing the infrastructure. Where possible, we used existing tools and software; however, significant software development and configuration was necessary to securely and accurately support the breadth of our courses. We recently completed and report here the final phase of this project, focusing on assessment of this learning tool; such structured assessment is still rare in the literature on remote laboratories. Specifically, we describe the project from three perspectives; students’ exam results, students’ lab reports, and students’ satisfaction with the distance experience (based on interviews), and applying an assessment framework based on triangulation between qualitative (interview based) and quantitative (survey based) approaches. We will also describe the difficulties we experienced in implementing this project and the techniques we used to solve them. PRIOR WORK Various researchers have examined the development and use of remote laboratories for engineering and computer science training [2, 4, 5, 10, 12, 13]. Some of these papers describe virtual laboratories (including simulations and virtual tools), while a few describe remote interaction with laboratory equipment through either remote control or remote monitoring. Other researchers have considered the operation of remote laboratories specifically in the context of networking or telecommunications [1, 6, 7, 8, 9, 11, 15, 16, 17, 18]. The application of remote laboratories is still a young and experimental field so that this work does not in general go beyond a cursory assessment of educational outcomes.
0-7803-9077-6/05/$20.00 © 2005 IEEE October 19 – 22, 2005, Indianapolis, IN 35th ASEE/IEEE Frontiers in Education Conference S3F-7
Session S3F Additionally, the focus is more on simulation or the creation of virtual environments and less on remote operation of actual networking equipment. Simulated environments are limited as they (1) generally do not fully emulate the real environment, (2) require considerable effort to maintain up to date environment, and (3) can fail to provide accurate systems perspectives. Among prior efforts, several instances bear specifically on our work. Baumgartner, et al. , describe a remotely accessible virtual router tool with which a student can explore the behavior of networking protocols. This approach provides a flexible platform but not one that mirrors typical real work installations, such as networks with specific vendors’ equipment. A goal of our labs is to provide students with actual experience in network elements typically found in a production environment. In , Hua and Ganz describe a web enabled remote laboratory environment, but do not include a mechanism for the remote user to reconfigure the network element connectivity. In , Saliah, et.al, describe a number of methods for resource management, some of which we have recreated in our environment. We followed a similar approach to ensure authenticated and authorized access while preserving the integrity of the students’ experiments. We also focused on methods to assist in the scheduling to prevent contention of resources. In , Nedic et al, provide a well thought out description of remote, virtual and real laboratories and the values and differences of each. We choose to apply all three methods (virtual, remote and real access) in our environment. In , Ogot et al, take a significant step toward assessment of a remotely operated and accessible mechanical engineering lab. This paper concludes that there is “no significant difference between the educational outcomes from students who performed the experiment remotely, versus those who carried out the experiment inperson,” a phenomenon we explored and duplicated in the context of remote networking education. APPROACH AND CAPABILITIES The ReLI project is building an infrastructure for telecommunications experiments that students can access through the Internet using conventional browser software. Initial efforts focused on: • • • •
A standard environment for developing computer assisted laboratory experiences by faculty and teaching assistants, An administration, authorization, and scheduling system to manage simultaneous access for multiple students, A security structure and policy that will protect the actual laboratory resources from unauthorized access and potential damage, A use of common browser interfaces, so that students are not required to purchase any software or hardware component in order to interface with ReLI. Sets of online reference material, to better help students that have no collocated TA supervision.
An ability to supervise student work through a monitoring mode. A virtual networking environment based on simulations.
During 2004 we implemented equipment interfaces to permit remote students to interact with real components in our lab from any remote location (via a personal computer) and to provide dynamic interconnection between lab components under student control. Students have the ability to dynamically select the laboratory exercise they want to do at a specific point in time. Teaching assistants have the ability to observe from the Interdisciplinary Telecommunications Program (ITP) Laboratory the activities performed by a remote student in real time. Students can also emulate labs at their command, by means of a drag-and-drop Network Designer, complemented with Network Simulator Software. It is expected that students will make use of these resources for introductory labs while they learn how to properly use real equipment. This software is robust against user mistakes and helps students build confidence. We are continuing efforts to develop fail-proof systems in order to prevent unintentional / intentional damage to equipment configurations and ensure availability. This includes equipment restoration and cleanup (i.e., removal of activities done by previous students so new students receive a pristine lab scenario). Lastly, we created user-to-TA communications interface to enable real-time support for Distance Students. We have also created extensive video streaming content, including: Class Recitation by Instructor: the instructor covers background and related topics as well as presents the structure of the lab to be covered during a specific week, available “on demand” to accommodate remote students’ schedules. Lab Development Lectures: we record each lab development as students on the campus perform them. The resulting online video laboratory takes remote students step by step across each lab objective using the same network elements they will be accessing remotely. The material assists both as a laboratory and exam study aid (without the need for contention for actual laboratory resources). Pre-Lab Lectures: additional video material for students with less technical background to assist in preparation for individual laboratories and thematic groupings of laboratories. Post-Lab Lectures: optional, in depth follow up and exploration video materials and laboratory experiments. Flash Based Tutorials: an introductory package that takes remote students through the setup of their remote stations so they can access ReLI resources. RELI ARCHITECTURE
At a high level, the ReLI architecture provides license controlled access to traffic generation (SmartBits) equipment, a router network and multiple computers via several protocols (Figure 1). 0-7803-9077-6/05/$20.00 © 2005 IEEE October 19 – 22, 2005, Indianapolis, IN 35th ASEE/IEEE Frontiers in Education Conference S3F-8
FIGURE 3 PICTURES OF THE RELI COMPONENTS
FIGURE 1 AN OVERVIEW OF THE RELI ARCHITECTURE
A more detailed depiction (Fig. 2) shows the use of connectivity matrices to enable student control of equipment interconnection; actual equipment is shown in Figure 3.
Table 1 describes the four methods that remote students use to access the different network components of the Telecommunications System Lab. TABLE 1
Connectivity methods available to the student. TECHNOLOGY
A free application for remote desktop access Provides access to our packet generators and sniffers Permits student supervision during labs
A non-intrusive PC management tool Taps into Mouse, Keyboard and Video cables Provides restart without loss of connectivity
LICENSE SERVER A low cost network simulation tool Loans software licenses temporarily Recovers them after lab completion CONNECTIVITY MATRIX
FIGURE 2 A DETAILED DEPICTION OF THE RELI ARCHITECTURE
A programmable layer 1 switch (script based) Permits on-demand equipment interconnection Student can run script to relocate network elements
ReLI implements an Ethernet Connectivity Matrix based on an APCON Physical Layer Switch (Figure 4). This switch has an Ethernet backplane that allows a transparent interconnection between any devices that use similar network technology; this equipment also features both a GUI and a scripting interface that permits users to select and program port interconnectivity. Originally designed for network security management, this switch was adapted to provide remote students the ability to interconnect network elements on command; remote students make use of pre-recorded scripts that interconnect the equipment on basis of the lab experiment being performed
0-7803-9077-6/05/$20.00 © 2005 IEEE October 19 – 22, 2005, Indianapolis, IN 35th ASEE/IEEE Frontiers in Education Conference S3F-9
Session S3F •
FIGURE 4 A DEPICTION OF THE CONNECTIVITY MATRIX
The tool connects to a server residing in our lab to retrieve the proper licensing information to unlock a student’s version. Distance students are required to install/store on their PC’s a 50 MB application.
Router Pod & IP KVM: • Lab functionality can be replicated via a combination of the IP KVM based PC management system in conjunction with access to a Router Pod. • A Router Pod consists of a Cisco Communication Server accessible online – with the ability to console into any other Cisco equipment. Students can interface and configure all network equipment via the Communication Server, and all PC’s via IP KVM.
EXPERIMENTAL LABS The major part of effort has been the creation of a repertoire of remote laboratory experiences to be used in the curriculum of the Interdisciplinary Telecommunications Program. These include 12 complete labs that focus on various aspects of data networking (Table 2). TABLE 2
Currently available remote laboratory experiments LAB #
1 2 3 4 5 6 7 8 9 Optional
LAN Cabling and Fiber Introduction to LANs Advanced Switching and VLANs Intro to Routers and Routing Protocols Advanced Router Configuration xDSL Technologies VoIP Technologies Wireless LANs ATM Technologies Frame Relay, ISDN DDR Web/FTP/DNS/DHCP/TELNET Server
As a specific example, we describe here Lab #4, Advanced Switching and VLANs (Figure 5). Objectives: • Introduce students to basic switch and router management configuration • Develop skills in manipulation of LAN switching technologies to accommodate multiple user/department scenarios on first one and then across multiple switches. • Configuration of Routers to perform inter-VLAN routing Network Simulation Software: • Remote students can download online a tool that allows them to design a network by means of a drag and drop interface. Students can choose among 30 different Cisco routers/switches/PCs, interconnect them at will, and then configure each equipment component using a console window (identical to a production environment GUI).
FIGURE 5 DEPICTION OF THE VLAN CONFIGURATION FOR LAB #4
ASSESSMENT The first step in creating an assessment program is to define the learning objectives for the class – the specific learning outcomes expected of a student. The high-level learning goal of the lab course is “to obtain practical experience that builds on the theoretical understanding acquired in other courses and to gain hands on experience with various networking gear”. The specific learning objectives of the original (in class) course included such items as: (1) the student will be able to evaluate the different factors that affect Layer 1, (2) the student will be able to evaluate Layer 2 performance and limitations such as collision and broadcast domain optimization, (3) the student will be able to design and optimize complex IP addressing, (4) the student will be able to implement routing protocols, such as RIP and OSPF and (5) the student will be able to evaluate and implement systems security. These are assessed by direct assessment methods, primarily based on quantitative standards of written and practicum examinations and lab reports, developed and stable over several years prior to implementation of remote lab capabilities . As we worked to assess our new remote lab experiments, we continued with direct assessment but found that some of what we sought to understand was qualitative in nature and required that we add indirect assessment
0-7803-9077-6/05/$20.00 © 2005 IEEE October 19 – 22, 2005, Indianapolis, IN 35th ASEE/IEEE Frontiers in Education Conference S3F-10
Session S3F techniques, requiring that we conduct interviews and surveys to assess the students’ experience . In 2004, we ran a pilot test of the remote infrastructure with one student. While obviously of limited statistical significance, the student’s results did exceed the average on all direct assessment measures, including grades from lab reports, practicum and written midterm examinations, a final project, and practicum and written final examinations. We also conducted unstructured interviews to assess the student’s experience half way through the term and learned (1) the areas where the student was experiencing difficulty (this included technical glitches and user interface problems), (2) the student’s initial perception of the remote lab experience and (3) the student’s perception of his preparedness for the midterm examinations. Following the end of the term (after grades were distributed), we conducted a structured interview, finding that (1) the student was far more tolerant of glitches than we expected, (2) the student was exceedingly pleased to have access to the labs in this manner, (3) the student intended to recommend the class to other students and (4) the technical glitches were tolerable if a teaching assistant was available to correct technical problems or advise the student on difficult concepts. In the subsequent term, we enrolled 5 remote and 7 inperson students in the lab class. We compared in class and distance students with direct and indirect techniques, specifically (1) the students’ lab reports and exam results and (2) the students’ experience based on interviews and surveys. While we would like to have more data points (i.e., students), we are presently limiting the numbers of students while we continue to refine the remote tools. I. The students’ lab report and exam results To provide a quantitative understanding of the effectiveness of the remote labs, we examined the lab reports, final project and exam results of both the in-class and remote students. These materials draw directly from the learning objectives previously described. The grading in this course is somewhat different from traditional lab classes in that there is no formal lab Grade Comparison In-class vs. Rem 100.0 90.0 80.0 70.0 60.0 50.0 40.0 30.0 20.0 10.0 0.0
II. Students’ experience based on interviews and surveys We began with unstructured interviews and used the results of these interviews to develop a set of survey questions. The interviews revealed fairly consistent perspectives among the students. The survey results in Table 3 show that students were generally pleased with the environment but had some concerns, specifically: (1) they appreciated the material made available through the video streaming, (2) they appreciated teaching assistant support, (3) they had complaints regarding the pace of the course and (4) there was variation in the perceived difficulty of the class. Differences between remote and in class student survey results were interesting and shed light on the problems that may still exist with remote laboratory experiences. Remote students found the teaching assistants far more helpful than the in-lab students. This might have something to do with the dependence the remote students might feel for the assistant. Also in-lab students thought that the pace was appropriate, whereas the remote students found it too demanding. From discussions with the teaching assistants, we found that the students initially spend more time on the labs than do in-lab students but this difference becomes small as the course reaches midterm. This may suggest that the remote experience lacks some feedback that the in-lab experience provides - possibly a function of the computer monitor experience; however, this difference erodes in a few weeks. TABLE 3
Student survey results (variance data noted in text)
Pr ac tic ot um e Pr ac tic um em
as s In -c l
em ot e
La b em Ex ot ec e ut La io n b Ex ec ut io n
In -c l
report. Rather, the instructors and teaching assistants grade mini lab reports, which provides details to such things as network configurations, ping tests and answers to specific questions (essentially the results section of a lab). The mid term and final exams consist of both a 2 hour written portion and a 2 hour lab practicum. The final project consists of an independent lab development exercise chosen by the student. The overall grading for the class breaks down as follows: 25% lab reports, 25% midterm exam, 25% project (data not yet available) and 25% final exam. As indicated in Figure 6, there was less than 5% difference between the in-person and remote student for lab reports and exam grades. The practicum grades showed an 8% difference, which causes us concern (we are now focusing on this issue). The variance was not listed on the graph due to the small sample size. Again, we recognize that the number of students was small in this sample, but this does provide us with a start in assessment.
Assesm FIGURE 6 GRAPH OF DIRECT ASSESSMENT MEASURES
Score (1-7 scale)
1. Teaching assistants provide adequate support 2. Pre-lab lectures provide valuable content 3. Lab facilities are NOT available when needed 4. You are pleased with remote access to the labs 5. Lab exercises are comprehensive 6. Video streaming content is valuable 7. Remote labs enhanced your education 8. Course pace is too quick 9. Remote labs are too difficult 10. Lab equipment generally works well
6.4 6.8 1.7 7.0 6.0 7.0 6.8 6.4 3.6 4.4
4.6 6.7 4.6 N/A 6.3 6.1 N/A 5.4 N/A 6.2
0-7803-9077-6/05/$20.00 © 2005 IEEE October 19 – 22, 2005, Indianapolis, IN 35th ASEE/IEEE Frontiers in Education Conference S3F-11
Session S3F We also conducted a follow up interview to find aspects not captured in the survey. We found the following issues: (1) technical glitches can result in hours of delay for the remote student, (2) improving the GUI by providing an easily abstracted view of connections would speed the completion of the exercises and (3) the ability to repeat labs is valued greatly. In addressing item (1), we have a teaching assistant always available for the remote student during their scheduled sessions. The students can (and often do) repeat the session, but this is without the teaching assistant. While we would like to work to resolve the technical glitches that arise, as anyone who works in a lab can attest glitches simply occur. We also plan to improve the network representation to the student through the use of clickable diagrams, which would provide consoles and similar access to the various devices. As pointed out in item (3), we found that remote labs may provide a benefit not captured by the in-person labs, the ease for in-class students to repeat an exercise remotely. In fact, we found in the earlier survey that the in-class students felt that the facilities were not available when they needed them. We see that the consistent reuse of the remote environment as a means of reviewing or reinforcing the material for both remote and in-lab students as being a strong motivator to continue to improve this environment. CONCLUSIONS Effective remote laboratory experiments in the area of networking systems education are feasible given the right investment in infrastructure. The result that direct assessment of outcomes is comparable between in class and remote students on exams and labs is encouraging and a little surprising given the “hands on” character of laboratories, but broadly consistent with the literature on other forms of distance education. However, indirect assessment via interviews and surveys suggests room for improvement and areas of frustration and perceived difficulty for remote students. Furthermore, the discrepancies we see in the practicum scores cause us concern and we plan to work to better understand the differences. We find that the field will benefit from further experimentation coupled with increasingly rigorous and repeatable assessment on which to build sound remote laboratory pedagogy. We recognize that this paper is only an initial step towards making more rigorous assessments, but now with the infrastructure in place, we should soon be able to provide more significant contributions. ACKNOWLEDGMENT This research was supported by The National Academy of Engineers Gordon Prize and (in earlier work) by the Colorado Institute of Technology (CIT). REFERENCES 
Diop, A., Berqia, A., Harms, J.: Remote Real Laboratory: Linux Installation and Configuration. International Conference on Internet Computing 2003: 232-236.
Shen, H., Xu, Z., Dalager, B., Kristiansen, V., Strøm, O, Shur, MS, Fjeldly, T.A., Lu, J., and Ytterdal, T., "Conducting Laboratory
Experiments over the Internet", IEEE Trans. on Education, Vol. 42, No. 3, pp. 180-185 (1999). 
Ogot, M., G. Elliott, and N. Glumac, “An Assessment of In-person and Remotely Operated Laboratories”, Journal of Engineering Education, Vol. 92, No. 1, 2003, p. 57-62.
Bourne, J., et al. 1996. A Model for On-Line Networks in Engineering Education. Journal of Engineering Education. 85(3): 253–262.
Bucciarelli, L.L., et al. 2000. ECSL/MIT Engineering Education Workshop ‘99: A Report with Recommendations. Journal of Engineering Education. 89(2): 141–150.
Nedic, Z., Machotka, J., Nafalski, A., Remote laboratories versus virtual and real laboratories. ASSE/IEEE Frontiers in Education Conference, Session T3E, IEEE 2003.
Fabrega, L., Massaguer, J., Jove, T., and Merida, D. A Virtual Network Laboratory for Learning IP Networking, Proceedings of the 7th Annual Conference on Innovation and Technology in Computer Science Education, June 2002.
Liu, S., Marti, W., and Zhao, W., Virtual Networking Lab: Its Concepts and Implementation, in Proceedings of ASEE Annual Conference, June 2001.
Kassouf, M., Pierre, S., Levert, C., and Conan, J., “Modeling a Telecommunication Platform for Remote Access to Virtual Laboratories,” in Proc. IEEE Canadian Conference on Electrical and Computer Engineering, Alberta, Canada, May 9-12, 1999, pp.127-132.
 Deniz, D., Bulancak A., and Özcan, G., “A Novel Approach to Remote Laboratories”, ASEE/IEEE Frontiers in Education Conference (FIE’03), Nov. 5-8, Colorado, USA, Session T3E, pp. T3E- 8-12, 2003.  Baumgartner, F., Braun, T., Kurt, E., Weyland, A.: Virtual routers: a tool for networking research and education. Computer Communication Review, 33(3): 127-135 (2003).  Hua, J., Ganz, A.,"Web Enabled Remote Laboratory (R-LAB) Framework", Frontiers in Education (FIE 2003), Boulder, Colorado, November 2003.  Gillet, D., Latchman, H.A., Salzmann, Ch. and Crisalle, O.D., “HandsOn Laboratory Experiments in Flexible and Distance Learning”, Journal of Engineering Education, April 2001, pp.187-191.  Saliah, H., Saad, M., Villardier, L., Assogba, B., Kedowide, C., Wong, T., " Resource Management Strategies for Remote Virtual Laboratory Experimentation ", 2000 Frontier in Education Conference , Kansas City, October 18-21, 2000.  White, B., Lepreau, J., Stoller, L., Ricci, R., Guruprasad, S., Newbold, M., Hibler, J., Barb, C., Joglekar, A., “An Integrated Experimental Environment for Distributed Systems and Networks”, In Proceedings of the Fifth Symposium on Operating Systems Design and Implementation (OSDI '02), pp. 255-270, December 2002.  Sung K. Yoo, Scott Hovis: Remote access internetworking laboratory. SIGCSE 2004: 311-314.  Esche, S. K., Chassapis, C., Nazalewicz, J. W. & Hromin, D. J. (2002). A scalable system architecture for remote experimentation. Proceedings of the 32nd ASEE/IEEE Frontiers in Education Conference, Boston, Massachusetts, USA, November 6 - 9, 2002.  Baumgartner, F., Braun, T., Kurt, E., and Weyland, A.: Implementation of a Distance Learning Module Based on Emulated Routers, KiVS03, Leipzig, Germany, March 25-28, 2003.  See “Beyond Confusion: An Assessment Glossary, Association of American Colleges and Universities” (http://www.aacu.org/peerreview/pr-sp02/pr-sp02reality.cfm).  "Qualitative Methods Used in the Assessment of Engineering Education," Journal of Engineering Education, Vol. 93, No. 1, 2004, p. 65-72, Leydens, J., B. Moskal and M. Pavil.
0-7803-9077-6/05/$20.00 © 2005 IEEE October 19 – 22, 2005, Indianapolis, IN 35th ASEE/IEEE Frontiers in Education Conference S3F-12