Open Source Cloud Computing Platforms. Thiago Cordeiro, Douglas Damalio, Nadilma Pereira,. Patricia Endo, André Palhares, Glauco Gonçalves,.
2010 Ninth International Conference on Grid and Cloud Computing
Open Source Cloud Computing Platforms Thiago Cordeiro, Douglas Damalio, Nadilma Pereira, Patricia Endo, André Palhares, Glauco Gonçalves, Djamel Sadok, Judith Kelner
Bob Melander, Victor Souza, Jan-Erik Mångs Ericsson Research Stockholm, Sweden {bob.melander, victor.souza, jan-erik.mangs}@ericsson.com
Federal University of Pernambuco (UFPE), Recife, Brazil {thiagocordeiro, ddamalio, ncvnp, patricia, andre.vitor, glauco, jamel, jk}@gprt.ufpe.br
Produced by Xen.org, the community project that maintains the widely known Xen5 Hypervisor, the Xen Cloud Platform (XCP) [10] copes with automatic configuration and maintenance of Xen-based clouds [8]. Eucalyptus is developed 6 by Eucalyptus Systems and is offered in two versions: an enterprise edition and a community edition. The community edition is open source and is capable of leading with a distributed architecture to manage virtual machines (VMs) using Xen and KVM7 hypervisors. The OpenNebula is similar to Eucalyptus in many aspects, but it strives itself for having a modular infrastructure and offering support to three widely used hypervisors, namely, Xen, KVM, and VMWare8.
Abstract — With the popularization of cloud computing, several enterprises and open-source communities have developed their own cloud solutions. A number of factors weigh on user selection, as each one has peculiar characteristics and may target different usage scenarios. Considering such challenge, this paper focuses on giving the reader an understanding of some major existing open cloud computing solutions – XCP, Eucalyptus and OpenNebula. Hopefully, a deep comparison of such solutions can leverage the cloud computing research area providing a good starting point to research groups and interested readers. Keywords - cloud computing; open source platforms; XCP; Eucalytpus; OpenNebula.
I.
We highlight these three solutions because of two issues. Firstly, the three solutions pertain to the class of Infrastructure as a Service (IaaS) cloud layer [1], which makes such solutions comparable together. Secondly, these solutions are among the most popular solutions in their category [11]. In addition, this paper also offers an extensive discussion comparing them under several aspects and offering appropriated scenarios for each solution.
INTRODUCTION
Currently, it is common to access content across the Internet with little reference to its underlying hosting infrastructure, formed by several content providers. The same technology providing such level of locality transparency also offers a new model for the provision of computing services, otherwise known as Cloud Computing. More than 500 enterprises have entered the business of cloud providers: Amazon and their Elastic Compute Cloud (EC2) 1 , Google´s App Engine 2 , Microsoft´s Azure Platform3, Salesforce and their Force.com4 among many others. The Open Source Community has also been very active in the Cloud Computing are with numerous contributions in related areas such as system and network virtualization. Moreover, the open source community is actively developing solutions to manage clouds, especially those employed in academic research around the world.
The paper is organized as follows: Sections II, III and IV, describe XCP, Eucalyptus and OpenNebula, respectively. The discussion about these three solutions is presented in Section V. Finally, Section VI summarizes our conclusions. II.
XEN CLOUD PLATFORM
The Xen Cloud Platform (XCP) manages storage, VMs and the network in a cloud. XCP does not provide the overall cloud architecture, but rather focuses on configuration and maintenance of clouds. It also enables external tools, including Eucalyptus and OpenNebula, to better leverage the Xen hypervisor [8].
Building a cloud environment for research purposes often involves choosing initially a solution for cloud management. This is a very difficult decision since each solution has specific characteristics adapted towards given scenarios. Previous papers ([7], [9]) surveyed open source solutions comparing features like their architecture, virtualization platforms and service type. However, the authors of both these papers used an extensive approach, giving succinct descriptions of each solution. In an attempt to give a more complete description of the platforms and their benefits, in this paper, we focused on three main open systems – XCP, Eucalyptus and OpenNebula.
A. Architecture Figure 1 shows the XCP architectural components [5]. Its basic component is the XCP Host, which is a Xen hypervisor enabled to communicate with other XCP hosts. Several XCP Hosts can be bound together into a XCP Resource Pool. A single XCP Host from this pool must be setup as the Master
1
5
2
6
http://aws.amazon.com/ec2/ http://code.google.com/intl/appengine/appengine/ 3 http://www.microsoft.com/windowsazure/ 4 http://www.salesforce.com/platform/ 978-0-7695-4313-0/10 $26.00 © 2010 IEEE DOI 10.1109/GCC.2010.77
http://www.xen.org http://www.eucalyptus.com/ 7 http://www.linux-kvm.org 8 http://www.vmware.com/ 366
of the VM infrastructure. The API calls make use of the XMLRPC protocol to transmit requests and responses over the network [6]. These XML-RPC requests and responses may also be exchanged between hosts in a XCP resource pool through HTTP protocol. If desirable, this inter-host communication can be turned secure using SSL-encrypted HTTP (HTTPS).
XCP Host, which offers an administration interface and commands others XCP hosts. Optionally, a Resource Pool may have a Shared Storage whose objective is to store and export VM images mainly for VM migration, which allows administrators to place and replace VMs on any XCP host. XCP Resource Pool
D. Summarizing XCP XCP is an open source infrastructure manager tool for clouds that does not provide the overall architecture for cloud computing, since it does not provide interfaces to end users to interact with the cloud. However XCP provides a useful environment for administrators and an API for developers of cloud management systems.
Private Network Shared Storage
XCP Host XCP Host XCP Host XCP Host XCP Host
III.
Figure 1. XCP Architecture [7]
B. Networking XCP networking deserves dedicated attention by itself. It is based on the Open vSwitch project 9 . The approach distinguishes virtual from physical interfaces while offering software based VLAN support. The Open vSwitch provides three software components: the Physical Network Interface (PIF), the Virtual Network Interface (VIF), and the Virtual Ethernet Switch (VES) 10 . PIFs represent physical interfaces attached on a XCP Host. Similarly, VIFs represent interfaces attached on the VM. The VES is a virtual switch on a XCP host, which can be used to connect VIFs with each other and with the PIF. A VES without an association to a PIF can be used to provide connectivity only between VMs on a given XCP Host, with no connection to the outside world (Figure 2).
Unlike XCP, the Eucalyptus architecture foresees two different user classes: administrators and clients. Once again reflecting a business oriented model view. The former are the users that manage the entire cloud, having access to all features of Eucalyptus. The latter are the final users that can request and make use of VM instances directly from Eucalyptus, without the need for administrators’ intervention. A. Architecture As presented in [2], Eucalyptus has four components, showed in Figure 3.
HOST 1
VIF To Physical Ethernet
CLC and Walrus Public Network
CC
CC
Cluster A
C. Inter-host communication XCP provides a management infrastructure with an appropriate API to install, monitor and manage various aspects
NC
NC
XCP supports VLANs through the use of additional PIFs corresponding to specific VLAN tags. Thus, it is possible to see all traffic on the physical network interface using a VES attached to a PIF associated with the interface.
Private Network
NC
Private Network
Figure 2. XCP Network Architecture
NC
VIF
NC
VM 2
PIF
NC
VES
NC
VM 1
EUCALYPTUS
Since its conception, Eucalyptus was designed to provide services compatible with Amazon´s EC2 cloud. Interestingly all communication is made through Web Services standards. Eucalyptus reflects interest in bridging open cloud software with that of enterprise solutions. This is better shown in the design of their external interface to be fully compatible with Amazon´s EC2 interface.
NC
Master XCP Host
Cluster B
Figure 3. Components of Eucalyptus Architecture [3]
VMs on a node are managed by a Node Controller (NC) running on this same node, allowing support of different hypervisors. NCs are grouped in clusters managed by the Cluster Controller (CC), which gathers state information from each NC, schedules client requests to individual NCs and manages configuration of public and private networks. At the
9
http://openvswitch.org/ In XCP documentation VES is called network. We use VES to avoid misunderstanding. 10
367
top of the architecture there is the Cloud Controller (CLC). It processes client requests and makes VM placement decisions. The Storage controller (Walrus) is a data storage service 11 compatible with Amazon´s Simple Storage Service (S3), which is responsible to manipulate VM images delivering them to NCs when a client instantiates a VM. A clear advantage of the Eucalyptus architecture is a natural fit for existing enterprise processing environments resulting from the hierarchical design.
When the CLC receives a runInstances request from a client, it determines which CC can support the VM querying CCs through describeResources operation and choosing the first CC that has enough free resources. In the same way, when a CC receives a runInstances request from the CLC it queries each NC through the describeResource message and chooses the first NC that has enough free resources. C. Networking Under Eucalytpus, a VM is accessed by the client through an external IP. VMs of the same owner can exchange traffic together, generating two traffic types: internal and external. In order to offer isolation between them, Eucalyptus deploys two virtual network interfaces of types: public and private.
Eucalyptus components have interfaces described by 12 WSDL documents. These interfaces can be separated in two groups: internal and external interfaces. The internal interfaces are used for communication between Eucalyptus’ components. The NC implements a number of operations to manage VMs. The main supported operations are: runInstance, terminateInstance, and describeResource. The first two are used to send commands to startup and stop a given VM, respectively. The last one reports current characteristics like memory and disk usage of the physical node. The CCs provide an interface described using WSDL, but one where nonetheless each operation can be used to manage several VM instances as indicated by the names: runInstances, terminateInstances, and describeResources.
The public interface can be used for communication between an owner and its VMs or between instances running in the same IP subnet [2]. On the other hand, the private interface is used for inter-VM communication across different subnets, handling the situation where two VM instances are running inside separate private networks but need to communicate with one another. These instances are connected 13 via the Virtual Distributed Ethernet (VDE), a process level implementation of the Ethernet protocol [2]. Figure 5 illustrates how these network components interact together.
The external interface is supported by the CLC that translates client requests operations to CC operations. Currently, for the sake of compatibility with Amazon´s EC2, Eucalyptus supports both SOAP-based and HTTP Querybased interfaces [3], whose main VM management operations are runInstances and terminateInstances. Maintaining compatibility with Amazon, Eucalyptus’ clients may only instantiate VMs from classes previously configured. Such classes specify the main aspects of the target VMs including memory size, disk size, operating system etc.
Physical Resource
VM A
B. VM Placement Figure 4 shows a sequence diagram of the entire process prior to running a VM instance triggered by a client request and finalizing when the VM is actually running in a node [2].
To Physical Ethernet
Physical Interface
Public Bridge
VM B
Public Interface
Public Interface
Private Interface
Private Interface
Private Bridge
Private Bridge
VLAN A
VLAN B
VDE
Figure 5. Eucalyptus Network Architecture (adapted from [3])
D. Summarizing Eucalyptus While working over Xen and KVM hypervisors, Eucalyptus provides an open source solution to manage the virtual infrastructure of a cloud. The use of interfaces based on Web Services is one of their key characteristics, allowing native integration with Amazon services. Moreover, the hierarchical architecture is designed to reduce human intervention. IV.
Figure 4. Sequence Diagram of messages to run a VM
OPENNEBULA
OpenNebula is a flexible tool that orchestrates storage, network and virtualization technologies to enable the dynamic 11 12
http://aws.amazon.com/s3/ http://www.w3.org/TR/wsdl
13
368
http://vde.sourceforge.net/
placement of services on distributed infrastructures [4]. A number of communities are actively using OpenNebula. Some 14 of these are: the European Space Astronomy Centre and the 15 European Organization for Nuclear Research (CERN).
and VMs are managed and monitored by the Host Manager and the VM Manager, respectively. The Virtual Network Manager (VN Manager) manages virtual networks by keeping track of IP and MAC addresses and their association with VMs. The SQL Database stores internal data structures.
A. Architecture OpenNebula has been designed to be modular in order to allow its integration with as many different hypervisors and environments as possible. It assumes that the physical infrastructure adopts a classical cluster-like architecture with a front-end, and a set of host nodes where VMs will execute. There is at least one physical network joining all the cluster nodes with the front-end. The front-end executes the main OpenNebula processes while the cluster nodes are hypervisorenabled hosts that provide the resources needed by the VMs.
Finally, the third layer is formed by modules called Drivers that supports different underlying platforms. These Drivers run on separated processes that communicate with the Core module through a simple text messaging protocol. There are drivers to deal with file transfers that are implemented by network protocols like NFS and SSH. Also, there are drivers to manage VMs that are dependent on each hypervisor running on the host. Finally, there are drivers to request services from external clouds like Amazon EC2 or ElasticHosts. B. Networking OpenNebula manages IP and MAC addresses of VMs and the virtual networks between them. There are two types of virtual networks: the Fixed network (Public) that uses a fixed set of IP and associated MAC addresses and the Ranged network (Red LAN) defined over a range of network addresses. VMs must pertain to one Red LAN and can, optionally, pertain to the Fixed network.
OpenNebula is designed with three layers in mind: Tools, Core and Drivers, as depicted in Figure 6. The Tools layer contains modules providing functionalities for administrators and clients. One component is the Command Line Interface (CLI) that can be used by administrators to manipulate the infrastructure through intuitive commands. The Scheduler module, responsible for VM placement, is implemented in this layer. Other tools can be created using the OpenNebula Cloud API which is based on a XML-RPC interface.
C. VM Placement As aforementioned, VM placement decisions are made by the Scheduler. The scheduler accesses all requests received by OpenNebula and, based on them, it keeps track of allocations and sends appropriate deployment or enforcement commands to the Core. The default OpenNebula scheduler provides a scheduling policy that places VMs according to a ranking algorithm, which relies on performance data from both the running VMs and physical resources. The basic operation of this scheduling algorithm is showed at Figure 7. The algorithm needs two inputs sent in the VM request: VM Requirements and VM Rank. The former consists of simple rules indicating the minimum resource requirements (as CPU or Memory), and the later consists of a policy for resource allocation. Thus, when a VM request arrives, the Scheduler, based on the VM requirements, filters those hosts that do not meet the requirements. After that, the remaining hosts are sorted using the Rank policy. Finally, the first host in this list is chosen to allocate the VM.
Figure 6. Components of OpenNebula (adapted from [4])
Similarly to Eucalytpus, OpenNebula works with administrative and client accounts. Administrators access OpenNebula through CLI, while clients launch and manage VMs using web services interfaces. OpenNebula implements an interface compatible with the EC2 Query API from Amazon and another one compatible with the Open Cloud Computing Interface from the Open Grid Forum16.
Algorithm 1: Scheduling algortithm. 1 2 3 4 5 6 7 8 9 10 11
The Core layer consists of components responsible for handling client requests and control resources. The main component of this layer is the Request Manager, which handles client requests through an XML-RPC interface calling internal components according to the invoked method. Hosts 14
http://www.esa.int/esaMI/ESAC/ http://public.web.cern.ch/public/ 16 http://www.ogf.org 15
Inputs: requirements, rank, hostsList; Outputs: selectedHost; Begin for each host in hostsList do if (host meets requirements) then candidates.new(host); end end sorted = sortByRank(candidates, rank); selectedHost = sorted(1); end Figure 7. Default scheduling algorithm
369
A careful adjustment of the inputs (Requirements and Rank) allows different placement schemes. TABLE I shows some policies that can be applied using OpenNebula. The first row indicates the selected administrative policy, the second indicates the heuristic used to achieve this goal, and the last row shows how the heuristic is implemented using the Rank attribute, which is used to sort hosts. Please note that, the administrator might achieve the same target using different heuristics, similarly to the ones showed in the second and third columns. TABLE I.
relates to the main purpose or expected use scenario of the cloud platforms. Analyzing OpenNebula and Eucalyptus, we can see that these platforms are focused on implementing an actual cloud by offering virtualized elements over a pool of physical resources. They implement, in fact, an IaaS that can be used as a public cloud. Though Eucalyptus can already deal with two different virtualization platforms, OpenNebula distinguishes itself as having a native ability provided by its modular architecture to deal with different underlying virtualization platforms supported by the use of its drivers, turning it much more flexible. In contrast to them, XCP focuses on providing a mere tool to manage a collection of virtualized hosts. In other words, XCP does not have a complete architecture to provide cloud services, though it can still be extended to do so.
EXAMPLE OF PLACEMENT POLICIES (ADAPTED FROM [4]) Policies
Target
Heuristic
Rank value
Packing Policy
Striping Policy
Load-aware Policy
Minimize the number of nodes in use Pack the VMs in the nodes to reduce VM fragmentation Rank must be set to use nodes with more VMs running
Maximize resources in a node Spread the VMs in the nodes
Maximize resources in a node Use nodes with less load
Rank must be set to use nodes with less VMs running
Use those nodes with more free CPU
B. Platform architecture The architecture of a platform tells a great deal about how that platform works and how it was built. It is also an important decision parameter when choosing an environment to implement a cloud computing system. Though a completely centered system is of course easier to implement and manage, it can suffer of performance and availability problems. On the other hand, hierarchical approaches on the control and management are more reliable.
D. Summarizing OpenNebula OpenNebula provides a modular architecture intended to be flexible. One of these modules is the Scheduler, an interesting algorithm that places VMs depending on their requirements. The client communication is also managed by modules that offer interfaces based on web services. It is perhaps the only open management platform that has invested into a tailorable VM placement algorithm. As such it may provide a nice environment for those researchers seeking to compare and develop different resource allocation strategies.
XCP can be seen as an example of a platform that has a centered architecture. The XCP Hosts are grouped on a resource pool which is single managed by the Master XCP host. In this aspect, Eucalyptus has a truly hierarchical approach. The cloud is divided into different clusters, each with its own cluster controller that is responsible for all nodes under that cluster. On the higher level of the hierarchy, the cloud controller is responsible for all the cluster controllers. OpenNebula works similarly to XCP with a front-end node and several hypervisor-enabled nodes. But, OpenNebula innovates in its completely modular architecture making it easy to integrate with different technologies. It has three main layers: Tools, Core and Drivers. Among the tools, we have the Scheduler, responsible for assigning new VMs to hosts. In the core we have different kinds of managers for controlling and monitoring VMs, networks, and hosts.
A limitation found with the OpenNebula is that, like XCP, their infrastructure assumes a classical cluster-like architecture with a front-end and without any redundant services. V.
DISCUSSION
In this section, we compare the open source cloud platforms presented earlier under different aspects. The main aspects and features studied here were chosen because of their believed fundamental importance on the decision of which solution a cloud operator should use. These aspects are: the main purpose of the cloud platform, the platform architecture, VM placement algorithm, client access interface, networking aspects, and communication protocols. At the end of this section, some use scenarios for each platform are discussed based on the differences pointed out here.
C. Virtual Machine Placement The placement algorithm of the platform is important to guarantee a clever and perhaps optimal use of the resources. Badly used resources always result in inefficiency. XCP does not offer a built-in option for automatic allocation algorithms. If a cloud provider wants to make it possible, he should implement it using the XCP API or use the separate Workload Balance software solution. The publically available Eucalyptus version has a simple algorithm that seems to pay little concern to VM placement on the cloud.
A. Main purpose Cloud providers should carefully choose a platform that copes with their objectives. As expected, different platforms provide different levels of service to administrators and clients. Therefore, the first feature discussed in this section
In contrast, OpenNebula brings an interesting approach to VM placement based on its defined Rank value, which can be used to implement useful policies. As such, it may be more
370
attractive to users who see resource usage and optimization as a primordial requirement.
In the second scenario, suppose we have a homogeneous pool of Xen or KVM hypervisors and want to offer a public cloud service to a community, more specifically offering a simple IaaS. In this case, the free version of Eucalyptus will fit very well, though it is not able to provide some important services like VM migration. Moreover, its allocation algorithms are simple and may not be efficient in some cases.
D. Client access interface This aspect concerns the problem of how cloud clients are going to access their leased resources. Public cloud systems should provide this interface insomuch the clients can operate their resources transparently. Notice this is an aspect that needs to be standardized hopefully in the future.
Now, consider that we have a pool of resources, possibly with heterogeneous virtualization platforms, and want to create a private/hybrid cloud over it. OpenNebula can easily support this. Also, suppose we do want to provide IaaS with more diversified and powerful functionalities, as virtual machine migration and a more useful resource allocation mechanism. Then OpenNebula can also provide this. Overall, OpenNebula seems to be more suited for research and experimental studies.
On OpenNebula and Eucalyptus, an external user is able to use their resources through a Web Service provided by the platform. These platforms are also compatible and similar with Amazon EC2. A further advantage for the OpenNebula is that it can easily be extended to implement other interfaces. Differently, XCP, limited by its design objectives, has no built-in way of making external users configure new VMs.
VI.
E. Networking aspects Cloud platforms must give attention to virtual networks, since they must ensure clients traffic isolation and help guarantying performance. Isolation is obtained through secure communication inside each different virtual network as well as the use of VLANs, whereas performance means that security should not make the communication inefficient.
CONCLUSION
In this paper, we presented a comparative description about three most popular cloud computing solutions – XCP, Eucalyptus and OpenNebula. The work also discussed differences about each one and described illustrative examples of use. It is hoped that by understanding some of the main differences between them, one may decide where and when each solution may be more appropriate for its use.
On Eucalyptus, each VM has already an explicit interface for both an internal virtual network and public access. Further, it allows the use of VLANs to separate different virtual networks contents. Similarly, OpenNebula have an actual separation between the internal traffic and the external access.
REFERENCES [1]
L. Youseff, M. Butrico, and D. Silva. “Toward a unified ontology of cloud computing”. In Grid Computing Environments Workshop (GCE08), 2008. [2] D. Nurmi, R. Wolski, C. Grzegorczyk, G. Obertelli, S. Soman, L. Youseff, and D. Zagorodnov. Eucalyptus: A Technical Report on an Elastic Utility Computing Archietcture Linking Your Programs to Useful Systems. University of California, Santa Barbara, October 2008. [3] D. Nurmi, R. Wolski, C. Grzegorczyk, G. Obertelli, S. Soman, L. Youseff, and D. Zagorodnov. The Eucalyptus Open-source Cloudcomputing System. In 9th IEEE/ACM International Symposium on Cluster Computing and the Grid, 2009. [4] OpenNebula Project. OpenNebula v1.4 Documentation. Available at: http://www.opennebula.org/documentation:documentation. Visited on May, 2010. [5] Xen.org. Xen Cloud Platform Administrator’s Guide. Release 0.1. 2009. [6] Xen.org. Xen Cloud Platform Development Kit Guide. Release 0.1, 2009. [7] P. T. Endo, G. E. Gonçalves, J. Kelner, D. Sadok. “A Survey on a Cloud Computing Solutions”. Workshop em Clouds, Grids e Aplicações, 2010. [8] Citrix Systems. “The Xen Cloud Project”, The Citrix Blog. Available at http://community.citrix.com/pages/viewpage.action?pageId=81690805. Visited on May, 2010. [9] B. Rimal, E. Choi, and I. Lumbi. “A Taxonomy and Survey of Cloud Computing Systems”. In International Joint Conference on INC, IMS and IDC, 2009. [10] Citrix Systems. “Xen Cloud Platform - Advanced Virtualization Infrastructure for the Clouds”, Avalable at: http://www.xen.org/products/cloudxen.html. Visited on May, 2010. [11] M. T. Jones. “Anatomy of an open source cloud”. IBM developerWorks. Available: http://www.ibm.com/developerworks/opensource/library/oscloud-anatomy. Visited on May, 2010.
XCP can also provide isolation by its use of VLANs to connect VMs that pertain to the same virtual network. But, differently from the other two platforms, there is no explicit separation between internal and public access interfaces. Note that an external access to a VM can also be implemented by using a new interface in a different VLAN. F. Communication protocol All platforms presented here use different inter-machine communication protocols. XCP uses XML-RPC as the intermachine communication protocol, while OpenNebula does make use of text based messages. Finally, Eucalyptus uses an approach based on the emerging web service standards. G. Scenarios With the main differences exposed we are able to discuss where and when each cloud platforms should be used. First, suppose we want just an easy way to manage a virtualized pool of resources. Suppose also we do not have any concern regarding providing an external service to a third party. In this case, a best approach would be to use XCP. Note however that in order to provide a complete IaaS using XCP, we would need to implement functionalities like VM placement algorithms and external interfaces.
371
