A Multi-layer Resource Reconfiguration Framework for Grid Computing Po-Cheng Chen
Jyh-Biau Chang1
Tyng-Yeu Liang2
Ce-Kuen Shieh
Yi-Chang Zhuang3
[email protected] [email protected] [email protected] [email protected]
[email protected]
Institute of Computer and Communication Engineering, Department of Electrical Engineering, National Cheng Kung University Department of Information Management, Leader University1 Department of Electrical Engineering, National Kaohsiung University of Applied Science2 Home Network Technology Center, Industrial Technology Research Institute/South3
ABSTRACT Grid is a non-dedicated and dynamic computing environment. Consequently, different programs have to compete with each other for the same resources, and resource availability varies over time. That causes the performance of user programs are degraded and unpredictable. For resolving this problem, we propose a multilayer resource reconfiguration framework for grid computing. As the named, this framework adopts different resource reconfiguration mechanisms for different workloads of resources. We have implemented this framework on a grid-enabled DSM system called Teamster-G. Our experimental result shows that our proposed framework allows Teamster-G not only to fully utilize abundant CPU cycles but also to minimize resource contention between the jobs of resource consumers and those of resource providers. As a result, the job throughput of Teamster-G is effectively increased.
Categories and Subject Descriptors C.1.4 [Parallel Architectures]: Distributed architectures; C.2.4 [Distributed Systems]: Distributed applications; D.4.1 [Process Management]: Threads; D.4.7 [Organization and Design]: Distributed systems; D.4.8 [Performance]: Operational analysis
General Terms Design, Experimentation, Performance, Measurement.
Keywords Reconfiguration, Non-dedication, DSM, Cluster, Grid.
1. INTRODUCTION Computational grid [10] is a new solution for data-intensive applications. This system is built by aggregating geographicallydistributed resources to form a single resource for resolving the same problems. It allows users to access different types of resources by a uniform mean and without caring about resource location. As a result, computational grids have been applied in
many research topics such as bioinformatics, high energy physics, weather forecast and data mining etc. However, different to computing clusters, grid resources are nondedicated. When users and resource owners simultaneously execute their programs on the same resources, the turnaround times of these programs will increase due to resource competition. This will degrade the willing of resource owners to share their resources with other people. Moreover, users will prefer to clusters rather than grids to be the platform of executing their jobs for the sake of program performance. Therefore, dynamic resource reconfiguration is necessary for resolving this problem. In the past studies, several methods have been proposed for the problem of resource reconfiguration. The authors of the paper [17] [23-25] proposed an immediate method to prevent the resource competition between users and resource owners. In other words, user jobs are immediately migrated to another resource when the resource owner starts to execute their jobs on their resource. This method is the best for minimizing the interference to owner jobs. However, it is not proper for user jobs to exploiting abandon CPU cycles of resources especially when the owner jobs just execute for a very short time. In order to effectively steal abandon CPU cycles, Pete proposed a method called linger longer [22]. As the named, a user job does not immediately migrate to another resource when owner jobs start to run. Conversely, the user program will linger at the same resources for a period time. If it predicts that the benefit from moving to another resource is more than the loss from competing with owner jobs for the same resource, it will decide to migrate to another resource. Although this method is better than the immediate method, it is not effective enough for exploiting available CPU cycles. The main reason is that the work load of owner jobs varies over time, and then the CPU cycles are not always fully exploited. Therefore, adjusting the percentage of stealing CPU cycles according to the workload of owner jobs is better than lingering at the same resource longer or immediately moving to another resource. In addition, the proposed methods are dedicated to intra-cluster resource reconfiguration. In other words, the source node and the destination node for job migration are located at the same cluster [1-3] [7] [12] [16] [18]. However, the resource demand of use jobs in grids increased so rapidly such that a single cluster can not provide sufficient resources for user jobs. As a result, the situation
that a user job is distributed across multiple clusters for parallel execution becomes common. Therefore, inter-cluster resource reconfiguration is necessary while it has not received enough attention. As above discussed, we propose a multiple-layer resource reconfiguration framework (MRF) for grid computing. As named, this framework provides three different-layer resource reconfigurations, i.e., virtual processor, node and cluster. Furthermore, the owner workload is classified info four different state, i.e., light-loaded, medium-loaded, heavy-loaded and pulseloaded. Different-layer resource reconfiguration is applied for different workload states, respectively. We have implemented the proposed framework on a grid-enabled software DSM system called Teamster-G [5-6]. Our experimental results show that MRF can allow Teamster-G not only to fully utilize available CPU cycles but also to minimize the interference to owner jobs. As a result, the job throughput of Teamster-G can be effectively improved. The rest of this paper is organized as follows. Section 2 introduce our experimental environment i.e., Teamster-G. Section 3 details the proposed resource reconfiguration framework. Section 4 discusses our experimental results. Finally, Section 5 gives a short conclusion for this paper and our future work.
2. Experimental Environment
Figure 1. The system architecture of Teamster-G Teamster-G is a grid-enabled software distributed shared memory (DSM) system [8] [11] [13]. This system is developed by extending Teamster [4] to exploit the services of Globus toolkit [21]. With the support of Teamster-G, users can make use of shared variables to write parallel programs in computational grids. Teamster-G adopts a two-level update protocol for maintaining inter-cluster data consistency in order to minimizing the cost of data communication. As shown in Figure 1, each cluster consists of a portal node and a set of normal nodes. When the normal nodes arrive at synchronization points such as lock release or barrier arrival, they will propagate their updates only to the other normal nodes and the portal node in the same cluster. Moreover, the portal node will propagate the updates made by the local nodes to the other clusters only when all of the local nodes arrive at a
barrier or the ownership of a lock is migrated from one of the local nodes to another cluster. When a portal node receives the update from the other portal nodes, it will propagate the update data within the local cluster for maintaining data consistency. Since the data updates made by the normal nodes are accumulated, the amount of inter-cluster data communication can be effectively reduced. On the other hand, Teamster-G adopts a two-level thread architecture for parallelizing user programs. This thread architecture consists of the kernel-level and the user-level. The mapping of the user-level threads to the kernel-level threads is many to many. Basically, a kernel-level thread is regarded as a virtual processor which is a unit to share CPU cycles. Therefore, we adjust the number of virtual processors used for a program to change the percentage of CPU cycles distributed to the program. Moreover, Teamster-G support user-level thread migration. This mechanism is useful for node reconfiguration and cluster reconfiguration in the proposed framework.
3. Multi-layer Reconfiguration Framework According to our observations, the resource workload of a node, which is defined as the executing jobs of resource owner, randomly varies from time to time. We can generally divide the resources workload info four different state. They are light-loaded, medium-loaded, heavy-loaded and pulse-loaded. When the resource owners execute light-loaded applications on the certain nodes, it means the resource owners only need a small quantity of computational ability of these nodes. The workload of this condition is too small to be worth a single glance; therefore, it is not necessary to take any action of resource adjustment. In another case, if the resource owners execute medium-loaded applications, it means the resource owners need a certain degree of computational ability of these nodes. However, the workload in this condition still remains some idle CPU cycles could be stolen. Consequently, if allowing the guest jobs of the grid systems linger on these nodes, the entire system throughput can be improved. In our MRF, we provide the virtual processor reconfiguration which can be adopted in this situation by deleting virtual processors and reducing their CPU utilization ratio. After the host jobs of owners finish their execution, the virtual processor reconfiguration can be adopted again to add back deleted virtual processors for increasing the CPU utilization. In the heavy workload situation, the resource owners have to execute some heavy-loaded applications. It means they would like to retrieve more computational power, and therefore considering about the autonomies of the resource owners, the grid systems can not but necessarily remove the guest jobs from the overloaded nodes to others. Therefore, our MRF provides node reconfiguration ability to hand off the overloaded nodes. Occasionally, the workloads of the resource owners are so heavy and result in that the whole cluster lies in heavy-loaded state. To reason by analogy, the grid systems compulsorily withdraw the guest jobs from the over-loaded cluster and our MRF provide cluster reconfiguration to remove the overloaded cluster. Finally, if the resource owners execute heavy-loaded applications in a very short period of time, this kind of workload is called pulse-loaded. Actually, this resource condition has almost no effect upon the guest jobs of the grid systems. Taking any reconfiguration action under this situation will induce unnecessary ping-pong effect that could be harmful to the performance of the
grid systems. Nevertheless, it is a troublesome matter that the grid system can detect this situation. In our MRF, the low cost of applying virtual processor reconfiguration could minimize the performance impact caused by the pulse loads.
4. MRF on Teamster-G 4.1 The System Architecture In this paper, we add six components of the MRF into our experimental platform, Teamster-G (shown as Figure 2). These components supply the user of the MRF a set of API to command corresponding reconfiguration manager for achieving different layers reconfiguration. In the node and cluster reconfiguration, two or more different managers need to work together. In the following paragraph, we are going to discuss the functions of these components respectively.
Figure 2. MRF on Teamster-G MM (Monitoring Module) is started on every node when Teamster-G is initiated. It is responsible for collecting workload information and providing this information to the user of MRF. Thus, the users of MRF can analyze this information to judge which layer of reconfiguration mechanisms should be performed. VPRM (Virtual Processor Reconfiguration Manager) is in charge when virtual processor reconfiguration mechanism is invoked. It has abilities to add or delete virtual processors, thus it can effectively modulate the CPU utilization ratio of Teamster-G node. NRM (Node Reconfiguration Manager) is provided with two abilities for user to perform node reconfiguration. These abilities include adding and deleting normal node in a Teamster-G cluster. In addition, CRM (Cluster Reconfiguration Manager) takes order from the user to produce cluster reconfiguration. Both NRM and CRM need helps form WTMM (Working Thread Migration Manager) and Coordinator. WTMM is responsible for moving the working threads of the DSM parallel application between nodes (as well as between clusters). Further, both NRM and CRM are also responsible for keeping the synchronization of working threads after thread migration. Coordinator is used to help NRM and CRM for finding a suitable node (as well as a suitable cluster) for substituting the overloaded node (as well as overloaded cluster) whenever the user of MRF wants to perform node reconfiguration (as well as cluster reconfiguration). However, in the current implementation, we do not discuss how Coordinator searches new grid resources because it is not the main concern of this paper. We assumed that Coordinator could find new grid resources by
querying the resources broker of Teamster-G. Furthermore, every Coordinator also collects the workload information of every node in the local cluster and exchanges this information with sibling Coordinator on the other cluster of the same Teamster-G Grid. Thus, if the user likes to perform node or cluster reconfiguration but there is no new Grid resources could be a substituted node or form a substituted cluster, Coordinator can find an appropriate light-loaded node or cluster to be substituted node or cluster from existed Grid resources.
4.2 The Operation of the MRF on Teamster-G In this section, we are bringing a living example to explain how of our reconfiguration framework operation. At first, it is a DSM parallel application which is running on a virtual dedicated cluster of Teamster-G with our MRF. The virtual dedicated cluster consists of many nodes. In a moment, certain nodes of this virtual dedicated cluster become overloaded due to continuously running applications of resource owner. In dealing with this situation, the user of MRF progress resource reconfiguration step by step with the multi-layer reconfiguration mechanisms rather than replace the overloaded nodes with new nodes immediately. In the beginning, the user of MRF periodically (the default value is every 30 sec. in the current implementation) inquires MM about the workload status of each node. MM finds the workload of every node is all in the light state, so it does not initiate any reconfiguration at this time. Awhile later, the resource owner starts to execute the host own job on a certain node. In the next workload monitoring state, the user of MRF is aware of workload change on the overloaded node via MM, yet it is in the medium state not heavy state. Therefore, the user of MRF just invokes VPRM to yield some virtual processors (the default value is cutting down a half of virtual processors) on the overloaded node for intra-node reconfiguration. After that, the resource owner again runs another host own job on the same node. It implies that the resource owner really demands a great amount of computational power of this node to do his (or her) jobs. Consequently, the user of MRF calls NRM to replace this node for intra-cluster reconfiguration after he (or she) detects that the workload state of this node becomes heavy. At this moment, NRM asks Coordinator on this cluster for a substituted node. First of all, Coordinator looks the resource broker of Teamster-G up for a new node. If there is no available node can be the substituted node then Coordinator finds a light-loaded node to be the substituted node for NRM. Finally, NRM commands WTMM to migrate working threads of the overloaded node to the substituted node and finishes node reconfiguration. Awhile later, the resource owner wants to execute a large-sized host own job. Once the user of MRF detects the average workload state of whole cluster is heavy, he (or she) decides to do cluster reconfiguration. The user of MRF firstly asks CRM to produce cluster reconfiguring. The CRM finds an available cluster to be the substituted cluster by Coordinator’s help. Coordinator performs finding a substituted cluster request by the similar way as finding a substituted node. After the substituted is found, the CRM cooperates with NRM to finish the request of cluster reconfiguration.
5. Experimental Results This section consists of two parts. The first part is the performance evaluation of intra-cluster reconfiguration with our work. In this part, we adopt our multi-layer reconfiguration strategy in three real applications and compare the performance of our method with that of other reconfiguration strategies. And another part is that we present a living example to observer the performance of intercluster reconfiguration with our work.
5.1 The Intra-cluster Reconfiguration In this part, we have implemented a set of application including MM (Matrix Multiplication), N-Body and SOR (Successive OverRelation) for evaluating the effectiveness of MRF. Our experimental environment consists of four SMP nodes, each node having four Intel Pentium III Xeon 500 MHz CPUs, and 512 MB main memory. These nodes are connected by a 100 Mbps Fast Ethernet and the operating system of these nodes is Linux Fedora Core2 for x86. The scenario of the intra-cluster reconfiguration is described as follow. In the beginning, we ran a test program, i.e., guest application, on Teamster-G with 4 nodes in one cluster. At the same time, we also ran a host application into one of the four test nodes. We adjust the problem size of the host application to induce different workload states, and then applied different resource reconfiguration, respectively. The experimental results are shown in Figure 3, 4, and 5. In these figures, WTM denotes that a half amount of work is migrated to another resource as long as the host application started to run. ND is to immediately move the guest application to another resource when the host application is executed. Finally, MRF is our method. Our method adapts to different host applications by applying different combinations of several reconfiguration mechanisms. No matter what kinds of host applications, we cut down a half of virtual processors which are occupied by guest application. When the workload states of host applications are recognized as heavy load, we remove the overloaded node from the Teamster-G cluster. Otherwise, we do not remove the overloaded node if the workload states of host applications are medium load or just pulse load. After host applications are finished, we will also add removed node and virtual processors back. We use throughput to be a metric of performance evaluation. The system throughput is defined as 2 / (the time spent on completing both of guest job and host job) .The system throughput of each method is normalized by comparing with that of MRF. (Heavy Workload) System Throughput 1.2 1
Figure 3 indicates the experimental result in the heavy host workload. ND has the least interference for the host application since the guest application is immediately moved to another node. Nevertheless, our MRF strategy can achieve the highest system throughput because the MRF strategy reconfigures resource progressively. In addition, the WTM strategy neither regulates the number of virtual processors nor releases the overloaded node. A half of working threads of the guest application still compete the computational power against the host application after reconfiguration. As a result, WTM is the worst for the case of the heavy workload state. Figure 4 shows the experimental result in the case of the medium host workload. In this case, our MRF strategy decreases a half of virtual processors which are employed by the guest application but does not immediately. Basically, the experimental result in this case is similar to the result in the case of the heavy host workload. However, MRF can improve about 25% system throughput compared to ND, if the experimental guest application is also part of medium loaded jobs, such as MM. the reason is that our MRF strategy can steal more available CPU cycles than ND strategy. Besides, as our anticipation WTM strategy can not perfume well. (Medium Workload) System Throughput 1.2 1
No Reconfiguration
0.8
ND
0.6
WTM
0.4
MRF
0.2 0 MM
N-Body
SOR
Figure 4. Experimental result in the medium workload
(Pulse Workload) System Throughput 1.2 1
No Reconfiguration
0.8 0.6 0.4
ND WTM MRF
0.2 0 MM
N-Body
SOR
No Reconfiguration
0.8 0.6 0.4 0.2 0
ND
Figure 5. Experimental result in the pulse workload
WTM MRF MM
N-Body
SOR
Figure 3. Experimental result in the heavy workload
Figure 5 shows that doing nothing is the best for the case of the pulse workload. Unfortunately, we can not recognize the burst workload unless we observe the workload state for a period of time. However, MRF is progressive in resource reconfiguration. It always applies virtual processor reconfiguration and then node reconfiguration. Therefore, after the burst workload is over, MRF
can increase the number of virtual processors and quickly regain the CPU cycles lost in the last virtual processor reconfiguration. As a result, the negative influence on system throughput can be minimized.
Host DSM parallel application (Heavy Workload) Executing Alone in G2 (Before Reconfiguration) Concuerrently Executing
According to our experimental results, we get a brief summary here. Although, the users who adopt our MRF can not obtain the best performance of all cases, the users can give consideration to different host workload situation.
Executing Alone in G2 (After Reconfiguration)
No Reconfiguration
5.2 The Inter-cluster Reconfiguration
MRF
In this experiment, we intend to evaluate the effectiveness of inter-cluster reconfiguration. Table 1 is the parameters of the experimental environment. Initially, the guest application is executed across the Cluster 1 and the Cluster 2. At the same time, we ran a host application in the Cluster 2. Because the host application requires a great deal of computational resources, all of the execution nodes in the Cluster 2 become overloaded. After applying MRF, the guest job distributed to the Cluster 2 is migrated to the Cluster 3 for execution. The experimental results are depicted in Figure 6 and Figure 7. Among them, the concurrently executing time includes the times of resource workload monitoring time (30 sec.) and the reconfiguration overhead. The results suggest that our multi-layer reconfiguration framework can achieve 7% improvement in system throughput compared with doing nothing. Furthermore, our multi-layer reconfiguration can deliver 5% and 53% performance improvement to the host application and the guest application, respectively. Table 1. Parameters of the experimental environment
Dedication
(sec.)
0
200
400
600
800
1000
Figure 7. The turn-around time of the host AP
6. Conclusions and Future Works In this paper, we have proposed a multilayer resource reconfiguration framework called MRF for grid computing. We have implemented the proposed framework on Teamster-G. Our experimental results show that MRF is more effective for minimizing the interference to the owner jobs, and maximizing the utilization of CPU cycles than the other methods. As a result, the job throughput of Teamster-G is significantly improved. In our current implementation, the effectiveness of resource reconfiguration is dependent on user decisions such as reconfiguration timing and adopted reconfiguration type. In order to reduce user’s work, we will develop an automatic mechanism to handle these problems for resource reconfiguration in computational grids.
7. Reference [1] Eduardo Hudo, R.S.M. & Llorente, I.M., 'The GridWay Framework for Adaptive Scheduling and Execution on Grids', SCALABLE COMPUTING: PRACTICE AND EXPERIENCE Volume 6, No. 3, pp. 1 – 8, September 2005. [2] Kaoutar El Maghraoui, B.S. & Varela, C., “An Architecture for Reconfigurable Iterative MPI Application in Dynamic Environments”, Proc. of the Sixth International Conference on Parallel Processing and Applied Mathematics (PPAM'2005)(number 3911 of LNCS, Poznan, Poland), pp. 258 – 271, September 2005.
The experimental DSM parallel application (Guest AP) Executing Alone in G1 + G2
[3] Reynaldo C. Novaes, P.R. & Cirne, W., “Non-Dedicated Distributed Environment: A Solution for Safe and Continuous Exploitation of Idle Cycles”, SCALABLE COMPUTING: PRACTICE AND EXPERIENCE Volume 6, No. 3, pp. 107 – 115, September 2005.
Concuerrently Executing
Executing Alone in G1 + G3 No Reconfiguration
[4] Ryan, J.P. and Coghlan, B.A. (2005) "Distributed Shared Memory in a Grid Environment" Proc.International Conference on Parallel Computing (PARCO 2005), Malaga, Spain, September, 2005.
MRF Dedication
(sec.)
0
100
200
300
400
500
600
Figure 6. The turn-around time of the guest AP
700
[5] Tyng-Yeu Liang; Chun-Yi Wu; Jyh-Biau Chang; Ce-Kuen Shieh; Pei-Hsin Fan, Enabling software DSM system for grid computing, Parallel Architectures, Algorithms and Networks,
2005. ISPAN 2005. Proceedings. 8th International Symposium [6] Tyng-Yeu Liang, Chun-Yi Wu & Ce-Kuen Shieh, “Teamster-G: A Grid-enabled Software DSM System”, In Proc. Fifth International Workshop on Distributed Shared Memory (DSM 2005), Cardiff, UK, May 2005. Held in conjunction with. CCGrid 2005, 2005. [7] Vadhiyar, S.S. & Dongarra, J.J., “Self Adaptivity in Grid Computing”, CONCURRENCY AND COMPUTATION: PRACTICE AND EXPERINCE Vol. 17, Issue 2-4, pp. 235 – 257, 2005. [8] Benny W.L. Cheung, C.L. Wang, and F.C.M. Lau, “LOTS: A Software DSM Supporting Large Object Space”, 2004 IEEE International Conference on Cluster Computing (Cluster2004), pp. 225-234, Sept. 20-23 2004, San Diego, California. [9] Eduardo Huedo, R.S.M. & Llorente, I.M., “A framework for adaptive execution in grids”, Software Experience and Practice Volume 34, Issue 7, pp. 631 – 651, June 2004. [10] I. Foster. & Kesselman, C., “The Grid2: Bluseprint for a New Computing Infrastructure”, Morgan Kaufmann: San Francisco, CA, 2004. [11] Ryan, J.P. and Coghlan, B.A. (2004) "SMG: Shared Memory for Grids" PDCS'04, Boston, November, 2004. [12] D.B. Weatherly, D.L., “Dyn-MPI: supporting MPI on non dedicated clusters (extended version)”, Technical Report 03003, University of Georgia, Jan 2003. [13] Gabriel Antoniu, Luc Bouge, Lacour Lacour, "Making a DSM Consistency Protocol Hierarchy-Aware: an Efficient Synchronization Scheme," CCGrid 2003, p. 516, 2003. [14] J. B. Chang, Tyng-Yeu Liang, Ce-Kuen Shieh, “A Transparent Distributed Shared Memory for Clustered Symmetric Multiprocessors”, accepted for publication in the special issue of The Journal of Supercomputing, September 6, 2003 [15] Nicholas T. Karonis, B.R.T. & Foster, I.T., “MPICH-G2: A Grid-enabled implementation of the Message Passing Interface”, Journal of Parallel Distributed Computing, pp.551-563, 2003.
[16] Lawlor, M.B. & Kale, L., “Adaptive MPI”, Technical Report 02-05, University of Illinois, 2002. [17] Frey J., T.T. & S, T., “Condor-G: A Computation Management Agent for Multi-Institutional Grids”, The Proceedings of 10th IEEE International Symposium on High Performance Distributed Computing, pp.55-63, 2001. [18] Gabrielle Allen, D.A. & Shalf, J., “The Cactus Worm: Experiments with Dynamic Resource Selection and Allocation in a Grid Environment”, International Journal of High-Performance Computing Applications Vol. 15, No.4, pp. 345 – 358, 2001. [19] I. Foster & C.K., “The Anatomy of the Grid: Enabling Scalable Virtual Organizations”, International J. Supercomputer Applications 15(3), 2001. [20] J. B. Chang, T.L., “Teamster: A Transparent Distributed Shared memory for Clustered Symmetric Multiprocessors”, The 2001 Workshop on Distributed Shared Memory on Clusters at IEEE CCGrid2001, pp. 508-513, 16-18 May 2001. [21] I. Foster. & C.K., “The Globus Project: A Status Report”, Proc. IPPS/SPDP '98 Heterogeneous Computing Workshop, pp. 4-18, 1998. [22] Kyung D. Ryu, Jeffrey K. Hollingsworth and Peter J. Keleher , "Mechanisms and Policies for Supporting FineGrained Cycle Stealing", In the 13th ACM International Conference on Supercomputing(ICS'99), (Rhodes, Greece June 1999) [23] R. H. Arpaci, A. C. Dusseau, A. M. Vahdat, L. T. Liu, T. E. Anderson, and D. A. Patterson, "The Interaction of Parallel and Sequentail Workloads on a Network of Workstations," SIGMETRICS. May 1995, Ottawa, pp. 267-278. [24] S. Zhou, X. Zheng, J. Wang, and P. Delisle, "Utopia: a Load Sharing Facility for Large, Heterogeneous Distributed Computer Systems," SPE, 23(12), 1993, pp. 1305-1336. [25] M. Litzkow, M. Livny, and M. Mutka, "Condor - A Hunter of Idle Workstations," International Conference on Distributed Computing Systems. June 1988 pp. 104-111.
