In other words, given a prece- dence tree (obtained from an optimizing compiler), how does one schedule a single but possibly compli- cated query involving ...
Scheduling Multiple Queries on a Parallel Machine Joel L. Wolf, John Turek, Ming-Syan Chen and Philip S. Yu IBM Watson Research Center P.O. Box 704 Yorktown Heights, NY 10598
Abstract There has been a good deal of progress made recently towards the ecient parallelization of individual phases of single queries in multiprocessor database systems. In this paper we devise and evaluate a number of scheduling algorithms designed to handle multiple parallel queries. One of these algorithms emerges as a clear winner. This algorithm is hierarchical in nature: In the rst phase, a good quality precedence-based schedule is created for each individual query and each possible number of processors. This component employs dynamic programming. In the second phase, the results of the rst phase are used to create an overall schedule of the full set of queries. This component is based on previously published work on nonprecedence-based malleable scheduling. Even though the problem we are considering is NP-hard in the strong sense, the multiple query schedules generated by our hierarchical algorithm are seen experimentally to achieve results which are close to optimal.
1 Introduction:
There has been a good deal of progress recently towards the ecient parallelization of the component phases of single queries in multiprocessor relational database systems. For example, [1, 2] have designed algorithms for handling the merge phase of a sort query in parallel. Similarly, [3, 4, 5] have designed algorithms for handling the join phase of either a sort-merge join or a hash join in parallel. These algorithms generally provide close to linear speedups in the numbers of processors utilized, usually even in the presence of skewed data [6].
0
Other queries, such as scans, and other phases of sort queries and join queries are actually easier to parallelize e ectively. These observations suggest that relational databases provide a setting highly amenable to the parallelization of individual query phases. When thinking of how to exploit this fact, two questions naturally arise: First, how does one handle more complicated intraquery parallelism? In other words, given a precedence tree (obtained from an optimizing compiler), how does one schedule a single but possibly complicated query involving multiple phases? How many processors should be allotted to each phase? Such a query might, for example, involve multiple scans, sorts and joins of a relatively large number of relations. Figure 1 shows a sample precedence tree for a complicated query. Second, how does one handle inter-query parallelism? In other words, how does one schedule a number of independently submitted queries, each of which may be complicated to one degree or another? How many processors should be allotted to each independent query?
1.1 Preliminaries
The literature on the subject of parallelizing queries is certainly in its infancy, with the focus so far on intraquery scheduling [7, 8, 9, 10]. In this paper we shall devise and evaluate a number of algorithms designed to handle both intra-query and inter-query scheduling. In fact, the winning algorithm for question 2 will be seen to be hierarchical in nature. In the rst phase, a good quality precedence-based schedule is created for each individual query and each possible number of processors. That is, we repeatedly answer question 1. This algorithm is based on dynamic programming. In the second phase, the results of the rst phase are used as input for the creation of a nonprecedencebased overall schedule. This is based on previous work
10
TASK EXECUTION TIME
12
11
7
8
9
4 2
3
5
6
1 NUMBER OF PROCESSORS
Figure 1: A Sample Query Precedence Tree on the nonprecedence-based version of the problem [11, 12, 13, 14]. Formally, the most general problem we consider in this paper is called the malleable scheduling problem, which can be de ned as follows. Consider a multiprocessor system consisting of P processors, and a set of N tasks which we wish to schedule on this system. We assume that each task j 2 f1; :::; N g can be allotted an arbitrary number of processors j 2 f1; :::; P g, and that its task execution time tj ( j ) > 0 is a nonincreasing function of the number of allotted processors. All of the processors allotted to a task are required to execute that task in unison. That is, these j processors are all required to start task j at some starting time, say j . They will then complete task j at some later completion time j + tj ( j ). The tasks are partially ordered by a precedence relation . A schedule will consist, for each task j 2 f1; :::; N g, of a processor allotment j , and a starting time j . A schedule is required to be legal in the following two senses: For any time , the number of active processors does not exceed the total number of processors. In other words, X j P: fj j
