On designing SNMP based monitoring systems supporting ubiquitous access and real-time visualization of traffic flow in the network, using low cost tools Ranganai Chaparadza
Abstract- SNMP is the widely used device specific monitoring method. With continued improvements in the performance of SNMP data collection and serving methods, SNMP-based monitoring is now more applicable to real-tine data collection, beyond traditional network management and trends gathering for capacity planning. Often, the user of an SNMP based monitoring system intends to make a selection of variables to be read over a duration during which monitoring is required, for example, just to get a snapshot of some resource utilization such as link utilization, traffic class utilization on some selected target routers when conducting network testing. In such cases, some kind of interactive monitoring system that provides the possibility to start and stop monitoring is often desired. The user of such a system requires an appropriate user interface through which selections of what to monitor can be made on-demand, and real-time traffic graphs can be displayed automatically. This paper describes how to design SNMP-based monitoring systems, supporting ubiquitous accesss and realtime visualization of traffic flow, using low cost tools. The concepts presented in this paper are backed by our implementation of a tool called SVM (SNMP-based Visualization Monitor) for QoS monitoring via access to QoS MIBs in DiffServ routers. Index Terms- SNMP QoS MLBs, Graphical User Interfaces (GUIs) for on-demand monitoring, real-time visualization of
monitoring data.
L INTRODUCrIoN
SNMP-based monitoring involves reading so called Object Identifiers (OIDs) implemented in MIBs by SNMP-
enabled systems such as routers, switches, servers and hosts, and presenting the data to the user in some way e.g. serving real-time graphs or OID values. An application that continues reading OlDs, uses the values in some computations and, presents the results to the user, is considered a monitoring application from its user's point of Manuscript received June 30, 2005. Ranganai Chaparadza, is with the Fraunhofer FOKUS Institute for Open Communications, 10589, Germany (phone: 00 49 30 3463-7102; fax: +49 30 3463-8000 e-mail:
[email protected]).
This work was funded by the Ministry of Education and Research of the Federal Republic of Germany together with Sienens Information and Communication Networks (ICN) AG Munich, as part of the KING Project. The author alone is responsible for the content of the paper.
1-4244-0000-7/05/$20.00 ©2005 IEEE.
view. Today, the networking research community is still developing and testing advanced mechanisms for QoS Routing, resilience, traffic engineering and distribution in IP based converged networks. In all these developments, researchers rely heavily on network monitoring to verify that each of the above mechanisms functions as expected when the mechanism is activated in the network. Today, vendors of routers, switches and other type of network devices are implementing very rich MIBs that enable the network operator/researchers to implement the above mechanisms and observe the effects of activating a particular mechanism in the network, by polling service-specific OlDs such as DiffServ traffic-class specific Counters or service specific statistics Counters implemented in the MIBs [13]. These are the kind of networking devices that researchers are using to build converged networks enhanced with the above mechanisms. Often, the researchers require a kind of SNMP based monitoring system that has a graphical user interface to display the topology of the network on the IP level and, provides the user with the possibility to select the links (IP interfaces) to monitor and visualize real-time traffic flow on the selected links. This is to limit the resources consumed by the monitoring and to limit SNMP traffic in the network, by selecting the links/nodes that need to be monitored over some time.
Given an appropriate GUI presenting the network topology to the tester and a visualization interface, the tester can select the links to be monitored on some node(s) in order to observe how some activated "traffic distribution" mechanisms on a particular node equally distribute the traffic to some destination using the set of egress links towards a number of neighboring nodes. Such testing involves stimulating the network with variable traffic load using traffic load generators that generate traffic mixes such as best-effort traffic, interactive real-time, signaling and control traffic classes, whose behavior need to be visualized. In this paper, "links" can be interpreted to mean layer-2 links or logical links (IP interfaces). Section 2 describes how to design such a monitoring system, using some selected low cost tools. Section 3 gives an example of such an SNMP-based monitoring system. Section 4 concludes on the viability and benefits of using the
868
links/nodes to the monitor back-end via a CGI script.
studied low cost tools and discusses further work on our SVM tool.
Availability of an API for the freely available NetSNMP tools [31 and availability of APIs for graphing tools [5], image production [61 and database connectivity for storing trends data[8][9].
II. USING SOME SELECTED LOW COST TOOLS FOR DEVELOPING SUCH A MONITORING SYSTEM, SUPPORTING UBIQUITOUS ACCESS AND REALTIME VISUALISATION OF TRAFFIC
C. Net-SNMP The monitor back-end can use the Perl interface to the freely available Net-SNMP tools [3] or the C based API for reading OlDs from the targets. Net-SNMP provides better security mechanisms that would ensure controlled access to the monitor.
A. Overview We consider development tools to be low cost when thinking in terms of availability of the development tools, development time, the amount of the resulting code and its adaptability and
maintainability.
D. An H7TP Server For ubiquitous access, an HTTP server [101 can be integrated with the monitor back-end. The HTTP server can be used to present the GUI through the user's browser for remote access to the monitor. From the GUI, the user can launch monitoring requests of selected links/nodes to the monitor back-end. The HTTP server also serves for serving dynamic content such as graphs to the user.
When designing the monitor, the developer needs the kind of development tools that provide among other features, the possibility to modify, both the user interface and the monitor code with ease whenever a need arises i.e. when graphical representations of new SNMP targets and OlDs need to be added to an SNMP-based monitoring system. The tools described below are useful to consider when developing an SNMP-based monitor and its suitable web-based GUI. Such a monitor will typically be comprised of the following integrated components: * A Back-end Monitoring Server that handles the monitoring of the selected OlDs i.e. QoS-traffic-class specific Statistics Counters for each known IP interface in the network. Each link on the network topology presented by the monitor GUI has to have some OlDs bound to it. * An HTTP server for remote access to the monitor GUI, serving dynamic HTML content such as 'traffic graphs to the user and for dispatching monitoring requests via the Common Gateway Interface (CGI), for example. * A GUI for presenting the network topology to the user so that the user can select target links or nodes to be monitored and launch monitoring requests to the monitor back-end. We demonstrate how the following tools can be used together to design such monitoring systems. B. Pern Programming Language and graphing libraries Perl [7] is a very good scripting language that is suitable for designing the Monitor Back-end server. Implementing the monitor back-end code in Perl is less costly in terms of development time and the amount of code than implementing in strongly typed languages such as Java. The following Perl features can be considered when developing the monitor back-end and a web-based GUI that interacts with it. * Support for CGI (Common Gateway Interface) for server side programming. The monitor GUI can interact with the monitor back-end using a CGI script via an HTTP server. The user launches a monitoring request from the GUI and the user's browser sends a list of user selected links/nodes to be monitored via a POST or GET method to the HTTP server part of the monitoring system. The HTTP server can then dispatch the request and pass the user's selected
E. JavaScript Programming JavaScript [11] is a scripting language for enhancing web pages and servers with interactive features in a variety of ways such as user input validation and dynamic web content. The monitor GUI can be designed as HTML code containing selectable elements (network links/nodes), with embedded JavaScript code for tracking user selected links/nodes using user events such as OnClick. The embedded JavaScript can play a very important role such as revealing the user's current selection, allowing the user to modify the selection and creating a formatted textual string that contains the user's selected links/nodes to be submitted by the browser to the monitor back-end server for monitoring, upon the "submit request" action of the user. JavaScript can also play a role in user input validation and dynamic loading of HTML pages from the monitor server via a client pull model. For example, when the user submits a request consisting of say five links, client side JavaScript code can be used to open a new window, pull an HTML page hosting five real-time graphs for displaying traffic trends concerning the selected links from the monitor server and, load the page into the newly created window for visualization. F. Macromedia Fireworks It is a tool for designing,
graphics [121. Designing
optimizing and integrating web web graphics and GUIs in
Macromedia Fireworks involves developing some web elements or the GUI as a single PNG image or multiply related PNG images and binding JavaScript behavior to some elements if required. The PNG can then be used to generate HTML code. These features can be combined to design both the monitor GUI and some static HTML pages containing JavaScript code for dynamically reloading embedded traffic graph images. Therefore, Fireworks can be used to design the monitor GUI as a PNG image with JavaScript bindings to an image map of each link or node on the network topology. Some HTML pages can also be designed as PNG images. For example, a page meant to
869
display five traffic images to the user can be designed with some five place-holder images, which will then be replaced by real-time images produced by the monitor back-end during the monitoring of some five arbitrarily user selected links. Client-side JavaScript can then be used to pull and display the appropriate HTML page based on counting the number of links in the user's request. The required HTML code is generated from the design PNGs. Other features of Fireworks to exploit are: * Image animation - In order to serve dynamic content such as real-time traffic graphs, client-side JavaScript can be generated or manually embedded into traffic display pages, to animate (swap) the images (graphs) at a rate of say one second, by modifying the source ("src") attribute of each image, swapping with a newly created image produced by the monitor backend. Client-side JavaScript can also be added to HTML traffic display pages, which then uses the "onLoad" event of a page to start the image reloading behavior, keeping pace with the monitor back-end. Only the images are dynamically re-loaded by the JavaScript from the monitor server and not the whole page. The effect is that the user sees real-time graphs all the time. *
Client side image maps together with some JavaScript can also be used for special buttons such as buttons for revealing the user's current selection and for clearing the selection register so that selections can be made afresh, using "OnC lick" events.
*
JavaScript Rollovers - Rollovers are used to change content based on the "OnMouseover" event. Rollovers can be used in highlighting the state of an element on the monitor GUI or for indicating whether selecting such an element for monitoring can result in a graph or say some logging of monitoring data.
Il1. AN EXAMPLE OF AN SNMP-BASED MONITORING SYSTEM THAT SUPPORTS UBIQUITOUS ACCESS AND TRAFFIC VISUALISATION The implementation of our tool called SVM (SNMP-based Visualization Monitor) for monitoring in our test-bed network called KING(Key Components of the Internet of the Next Generation), consisting of DiffServ routers, demonstrates how to use the low cost tools described in section 2. SVM is made up of several inter-working components: * The Back-end monitoring server, * An HTTP server and CGI (Common Gateway Interface) scripts, * HTML pages for serving dynamic real-time graphs of the flow of traffic belonging to different traffic classes in the network, * A Web-based GUI front end. A. The design ofthe SVM GUI The SVM GUI was designed using Macromedia Fireworks [12] and HTML code suitable for ubiquitous web
870
access to
the GUI was generated from the PNG design
images. Figure 1: shows the design of the SVM GUI as a network topology, designed as a PNG image and illustrates how the GUI looks in the user's browser when the corresponding HTML code is loaded from the server by the user's browser. The GUI is meant to display the network topology with its logical links between nodes and to provide buttons for the user to view his currently selected links, cancel the selection and re-select some links to be monitored and send(submit) the monitoring request to the back-end
monitoring server.
Figure 1: The SVM GUI for monitoring our
test-bed network The network nodes in the topology are routers named after some cities in the United States. The links between the nodes are logical interfaces (IP-level). The network displayed above serves as a testbed for testing advanced QoS-handling mechanisms such as traffic distribution/engineering mechanisms, policy-based admission control and routing and, resilience mechanisms. For testing purposes, the SVM is useful for visualizing the flow of traffic belonging to different traffic classes and link utilization on selected links in the network after a specific mechanism(s) has been activated. Image maps were created for the target elements namely a node e.g. Dallas and a link e.g. ChicagoAtlanta and for the buttons ("Show My Current
Selection", "ReSelect
Links" and
"Show Traffic"). Figure 2: illustrates image maps for buttons and selectable target elements (a single link or a node). Figure 3: shows how the user interacts with the monitor GUI to view the currently selected links.
ReSelect Links
functions to convert between hashes and arrays. For example, each target router and its associated data such as its link(interface) names and traffic classes configured on the device can have an a hash defined, whose keys are link names, traffic class names etc. After loading the target MlBs
Chic-a-
4E.-. %- ..- ...):
Figure 2: Image maps for buttons and selectable target elements (a node or single link)
Figure 3: Viewing the current selection
JavaScript code bindings were added to the image maps for the target elements so that when the user clicks on an element, a node or single link, the JavaScript binding uses "Onlick" events to register the selected element into a register that stores the elements selected by the user. Similarly, JavaScript bindings were also added for the button "show my current selection" so that the user can view the selected links at any time, also for the button "ReSelect Links" so that the user can cancel the current selection and restart the selection process and for the button "show traffic" so that the user can submit the monitoring request to the server. JavaScript binding to the "'show traffic" button (submit) uses the GET method to submit the request to the server and, the JavaScript then opens a new window for pulling real-time traffic images from the monitor back-end. The loaded HTML pages contain embedded JavaScript for refreshing the traffic images in real-time by reloading every image embedded in the page every second. B. The design of the back-end monitoring server The monitoring server, implemented in Perl [7], requires some input files such as a file that specifies the IP addresses of the target routers, their names and their associated SNMP community strings. Perl provides some possibility to create complex nested data structures on-demand such as anonymous hashes, hashes of hashes, arrays etc, with
the SVM server creates such hashes for nodes, where keys are node names e.g. Chicago. The use of Perl's foreach function allows the SVM to iterate over each node and do some tasks such as walking the ifDescr [2][13] column of the if Table [2][131 in the target MIB using the SNMP getnext [31 operation in order to search for the interface names and obtain the interface index values(i fI ndexe s of the interfaces, walking the [2][131) TrafficClassName [131 column to get the index values [21[13] assigned to the traffic classes and creating the following data structures for each node: interfaces(links) hash-table whose keys are interface names, an SNMP VarList [3] (variables list) object containing target OlDs for each traffic class configured on each interface, anonymous arrays for each traffic class on each interface for storing computed traffic transfer rates, anonymous arrays for each interface for storing time ticks, anonymous hashes for each interface with keys referencing data associated with the interface(link) such as the SNMP session object, traffic class VarLi st objects etc. The SVM finally creates a hash whose keys are interface names and the keys reference all the data required for monitoring the interface (link) such as the node of attachment, traffic classes etc. The use of anonymous hashes and arrays allows one to work with references, which can be stored as scalar values, which in turn can be used to create hashes of hashes or hashes of arrays with arbitrary nesting. The SVM's main hash table allows it to use and create some data associated with a particular link during the monitoring of the link. The main hash table is a hash of nested hashes and is four levels deep. The link names are actually keys to the main hash table. Such a design allows the SVM to know which link or links to monitor when a request arrives because the user's request submitted by the user's browser contains names of links, which are actually keys into the main hash table. The SVM does the sampling of the traffic class specific OIDs(counters) including the timestamp from the target, computes the transfer rates and then produces graphs using the GD [5] and related libraries for the target links every t seconds (sampling rate), where t is configurable, i.e. 1 second. The graphs are the traffic images loaded every second by the JavaScript embedded in the HTML display page. If errors occur during the reading of OlDs for a link an alarm image is produced in place of a traffic image so that the user can notice this error (OID reading failure) in the HTML page displaying the traffic images. The SVM accesses Juniper's ERX QoS statistics MIBS implemented in the nodes. The performance of the Perl based SVM monitor back-end is satisfactory enough to handle the demands of high sampling rate of approx. 60 OlDs, corresponding to about 15 links, in one second, though it employs sequential OID readings instead of parallel reading using multiple
871
threads
[4].
Upon
the submission
JavaScript C.
or
appropriate
How the SVM Tool works
The
user can
several links
select links to be monitored or
one
by clicking
several nodes. The
or
user
flexibility
to
requesting
the monitor-backend to monitor and
traffic
the
on
HTTP
view
requested link(s).
Server
that
(RequestReceiver.pl)
submiitted by containing
a
the
the
core
monitoring mww.
tetw t
Figure
dumps
and
dynanmically
e
the
CGI
script
every
new
request
a
formatted to
the
string
"Hot-file"
Figure
b
counting is
a
4: illustrates how
page,
result of the
of
an
u ser's
consisting
of five
utilizationi)
of
including
the
page contains
cg_re
HTML page
pull
precision
traff-ic
In
rates and
generated [14]
tools
8
control,
best-effort,
order
(reloads) the
determine
measurement
high
constant to
to
verify
transfer
bit the
rate
line-speeds
traffic
shape
m
VsIEL. L L.u I~~~~~~~~~~~~~~~~~...
.:::-;:
:. -.:
.:
., .: :.-
27.
;:. .
-:
..
6.0
....;
:.: ...
I:.:
::: ".
..:!.
.., .i
;.-
Figure 5: Dynamic Images of Traffic flow and per-Traffi'c Class Link Utilization
872
..:
I -.:
:::
using
and
flow rate.
Se.ttlabalIas~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~'
the
of traffic
...............
:..:
tl1e",
levels(link
4: Architecture and components of the SVM
'----
5
server as a
trend
and
by
Figure
node "S e at
the
which refreshes
polling-based
sampling
server
user.
from the
an
images.
with seconds-ticks. The HTML
second.
every
the
classes,
real-time
timestamps,
RUDE/CRUDE
computed
traffic
three
of SNMP
r
pulled
images showing
traffic
from the
by
pulls
button
the
to moni tor the
request
JavaScript code,
images
the monitor, the
to
displays
which then
interactive
'flow rate at high i.e. 10OMbitIs,
cDM
request
a
"Show Traffic"
knows which page to
we
f*w
of
the
to
the number of links selected
snapshot
traffic
inter-work.
stJ9, "~m u
before
display
pass every request to the
process.
monitoring components A Wet
selection
requested links
list of the
has the
the
to
dispatches
which
JavaScript one
The request is send to the
it
passes
browser
selection.txt used to back-end
change his/her
and
bound
the
CONCLUSION
In this paper, we demonstrated how some selected low cost tools, some of which are free can be used together to develop SNMP-based monitoring systems supporting user access control, ubiquitous access to the monitoring server and the visualization of real-time monitoring data. Such systems are required by researchers developing network mechanisms for QoS routing, resilience and traffic engineering/distribution, provided that the routers used to build the test-bed network implement rich Stat istics Counters that can be polled. In most cases, seeing the traffic changes per traffic class on some selected link(s) in the granularity of say one second is enough to see the effects of having a particular mechanism (feature) active in the network. Our experiences with Juniper's ERX router and its support for creating virtual routers in one box in order to build IP networks, showed that one can query a single virtual router for 60 OlDs (outBytesForwarded Counters and the SysUpTime), corresponding to about 15 links (IP interfaces), in one second, while the virtual router is transferring traffic at a relatively high line rate i.e. 10OMbit/s. The implementation of the SVM tool presented in this paper demonstrates how development time can be significantly shortened by using the low cost tools described in this paper. Future work on the SVM will involve making the tool more generic to support monitoring arbitrary OlDs, with the ability to classify OlDs into those for which graphs can be produced and those for which only logging can be done. We also intend to investigate how tools like network topology discovery tools could be used together with web
design tools such as Macromedia Fireworks for automated network topology drawing, with the possibility to edit the resulting network topology image as PNG design images in GUI design tools such as Macromedia Fireworks. REFERENCES [I]
D. Perkins, E. McGinnis, Understanding SNMP MIBS, Prentice Hall,
1996. [2] Douglas R. Mauro and Kevin J. Schmidt, Essential SNMP, O'Reilly
& Associates, 2001 [3] Net-SNMP Home Page: http://www.net-snmp.org [4] Robert Beverly 'RTG: A scalable SNMP Statistics Architecture for Service Providers", MIT Laboratory for Computer Science,
http://www.mit.edu/-rbeverly/papers/rte-lisaO2.pdf
GD graphics library home page: http:/lwww.boutell.com/gd/ [6] libpng: http:/lwww.libpng.org/pub/png/Iibpng.html
[5]
[7] Perl Programming Language: http://www.perl.com/ 18] Perl API for MySQL Open Source Database: http://www.mysql.com/ [9] Perl API for PostGreSQL Open Source Database:
http://www.postgresql.org/ The Apache H'lTP Server: http://www.apache.org 111] JavaScript Programming: articles from httpq/Hwww.javascnipt.com/
[101 1121
http://webreference.comijs/ Macromedia-Fireworks:
http://tww.macromedia.comi/.software/fireworks/ [131 Network Management Guide to the ERX 700/1400 [141
873
edge routing switches: Juniper Networks RUDE/CRUDE: UDP traffic generator and receiver tools
http://rude.sourceforge.net/
