Fault tolerant sorting network - CiteSeerX

Fault Tolerant Sorting Network Shay AssaP

Eli Upfalt

Abstract

that perform correctly in the presence of a significant number of faulty components.

We present a general technique for enhancing the reliability of sorting networks, and other comparator based networks. Our technique converts any network that uses unreliable comparators to a fault tolerant network that produces the correct output with overhelming probability, even if each comparator is faulty with some probability smaller than 1/2, independent of the other comparators. The depth of the fault tolerant network is only a constant time the depth of the original network, the width of the network is increased by a logarithmic factor.

This paper presents a general technique for enhancing the reliability of networks built of unreliable comparators. We demonstrate the novel technique through the construction of fault tolerant sorting networks. We comment on applications of our technique to other comparator based networks, such as merging and selection networks, in the last section.

1.

A comparator is a 2-input, 2-output device capable of sorting two elements. A sorting network consists of a collection of registers that pass through a number of levels of comparators. Registers store input elements we wish to sort, comparators connect pairs of registers. When two registers pass through a comparator the two values entering via the registers exit in sorted order. No more than one comparator is connected to any register per level, the number of levels is the depth of the network. A network is a sorting network if any set of elements entering the first level of the network exit in sorted order in the last level. An asymptotically optimal O(1og N ) depth network was first given by Ajtai, Komlos, and Szemeredi [AKS83].

Introduction

One of the major problems in large scale systems is the inevitable presence of faulty elements. The larger the system the larger the probability that some fraction of the system will fail to operate correctly. One way to overcome this difficulty is to design algorithms *Department of Applied Mathematics, The Weizmann Institute, Rehovot, ISRAEL. tIBM Almaden Research Center, San Jose, California; and Department of Applied Mathematics, The Weizmann Institute of Science Rehovot, ISRAEL. Work at the Weizmann Institute supported in part by a Bat-Sheva de Rothschild Award; and by a Revson Career Development Award.

CH2925-6/9010000/0275$01.OO Q 1990 IEEE

Yao and Yao [YY85] were the first to study sorting networks with stochastic faults. They assume a weak model of faults in which faulty comparators directly output the inputs with'All logarithms in this paper are taken to base 2.

275

out comparing them. A comparator in that model never changes the order of a pair that is already sorted. A deterministic version of this fault model was studied by Rudolph [Rud85], and Schimmer and Starke [SSSS].

tation. All registers store only copies of the input values. To use the extra registers, the network can copy values from one register to another. We measure the width of a network by the maximum number of registers used by the network in one level.

In the current work we consider a stronger fault model in which a faulty comparator outputs the two input values in an arbitrary order (or even outputs one of the input values in both outputs). Such faulty comparators can destroy the correct order among a list that is already sorted. Our goal is to construct a fault tolerant network, for the strong model of faults, that computes the correct sorted list with high probability, even if each comparator is faulty, with some fixed probability smaller than 1/2, independent of the other comparators. In other words, we require the fault tolerant sorting network to compute, with high probability, the correct output even if a constant fraction of its comparators, chosen randomly, are faulty.

The problem of noisy transmission of data has been extensively studied in the literature, and can be handled efficiently by error correction code [MS77]. In the current work we concentrate on faulty comparator and assume no faults in the copy and transmission of values in the network. Our main result is a general technique for converting any sorting network to a faulttolerant sorting network. Given a depth O(d) network for sorting N elements, our technique constructs a sorting network of depth O(d) and width O(N1ogN) that computes the correct output with probability 1 - 1/N even if each comparator is faulty with some fixed probability smaller than 1/2, independent of the other comparators.

Sorting networks usually use N registers, one for each input. The following argument shows that no N register network can achieve our goal. Let X1 denote the register that outputs the smallest element in the list. Clearly XI is compared at least once in the network, let X denote the last register to which XI is compared. If there are no more than N registers in the network, either X1 or 2 does not store the minimum value when they are compared. If that comparator is faulty, the network does not output the correct order. Thus, the failure probability of a fault tolerant network that sorts N elements with N registers cannot be smaller than the failure probability of each individual comparator.

The study of constructing reliable systems from unreliable components goes back to the work of von Neumann [VN56]. See also Pippenger [Pip851 for recent results. In these works the computing power of Boolean circuits is used in order to improve the reliability of circuits with noisy gates. Our new result shows that using comparators alone one can achieve similar results in the context of comparator based networks. The crux of our method is an expander based component that uses comparators and replaces the 'counting component' used in the construction of reliable Boolean circuits with noisy gates. While the comparator based component can not count, we show that it is powerful enough for enhancing the reliability of the sorting network.

To obtain a smaller failure probability we allow the network to use more than N registers. The network has N marked registers that receive the input elements and output the sorted list, the other registers are for internal use through the intermediate compu-

276

2.

2.1.

2.2.

The Fault Tolerant Sorting Network

Basic Components

The majority-expander and the majoritypreserver components are build of two basic components: the enforced comparators, and the stochastic-halver.

Overview of the Network

Definition: A fi-enforced-comparator is a construction with two inputs and two outputs that computes the maximum and minimum of the two inputs with error probability bounded by fi.

The network has two main parts. The first part simulates the comparators of the original sorting network. Through this part, each register X ; of the N registers of the original network is simulated by a set X;= { X : , ...,X l } of m = log N registers.

Theorem 1 For every 0 < p” < p < 1/2 there is a construction of depth and width 2( 2)1/(1%23/18) 3-6p log 1 that is a @-enforcedcomparator even if Peach of its comparators is faulty with some probability bounded b y p < 1/2.

We say that a register X: stores the correct value at a given stage of the computation if it stores the same value as the register Xi of the original network at the corresponding stage of the original network. For the correction of the computation we require that at each step of the computation all but a fixed fraction of each set X;stores the correct value. When register X; is compared with X j in the original network, our network compares each X : with X;, = 1, ..., m. Since a fraction of the comparators might be faulty, each comparison might increase the fraction of wrong values. In a bounded number of stages one cannot count or take majority. The heart of our simulation is a constant depth majorityezpander component that reinforces the majority in each set X;,after simulating each comparison of the original network.

Proof: We compute the maximum of X and Y by iterative use of the operator , Y]]. max[max[X, Y ] max[X,

Assume that we have a construction Mj that has depth D;,width W;, and computes the maximum of two elements with failure probability bounded by p ; . Plugging this construction into the above operator (see figure 1) gives a network that has depth 2D;, width 2W;, and computes the maximum of two elements with failure probability p;+1 I (1 - Pi)Pt

+ Pi(%; - P3) = 3Pt - 2P;

3

(1) The failure probability of the comparators is bounded by p . Thus po = p , and Do = WO = 1. Let E; = 1/2 - p ; . By plugging p; = 1 / 2 - ~ ; in (1) weget €;+I = ~ ; ( 3 / 2 - 2 ~ : ) . Ifpi 2 1/6, then E; I 2/6, and E;+]. 2 ~ ; ( 3 / 2 2(2/6)2) 2 23/18ei. Thus, if po 2 1/6, after 1 1 ‘tl 2 2/6 tl = log23/18 log 3(1/2-p) or Pt, I 1/6. If pj 5 1/6 then p;+l I 3 ~ : . Thus, pj+tl I 321-’(1/6)21 5 (1/2)2J. Let t 2 = loglog(2/fi) then Ptl+tZ I fiP.

The second part of the network generates the output. The input to this part are N sets, each with m = log N values. With high probability each set contains only a small fraction of values that are not the correct value for that set. The goal of this part of the network is to reduce the m values of each set to one correct value. This goal is achieved by successive use of a majority-preserver component.

277

Definition: An (e, 7,6, F)-stochastichalver is a construction with 2.t inputs and two sets of .t outputs. If the input set 82represents a value V , 01,02 < 1/2, such that no more than 2011 input values are smaller than V , and no more than 202& input values are larger than V , then, with probability 1-fi, the upper output set has no more than q l values larger than V , and no more than (281+ S ) l values smaller than V , and the lower output set has no more than 71 values smaller than V , and no more than (202 + 6)l values larger than V .

Using a similar construction we can compute the minimum of X and Y with probability $12. Thus, we get a construction that computes the maximum and the minimum of X and Y with error probability $. The depth and width of the whole construction is bounded by

+

To use this construction we need 211+t2 copies of each of the two inputs. These copies can be constructed in depth tl t 2 . 0

+

The construction of the majority-expander is based on explicit construction of expander graphs. Let G = ( A ,B, E ) denote a bi-partite graph, and let r ( X ) denote the set of neighbors of a set of vertices X , i.e.

Theorem 3 For every fixed 0 < 7 < 1, and 0 < S < 1, and for every 0 5 i 5 logm, there is an explicit construction with depth c1 log and width c2m/2’, where c1 and c2 depend on 7 and p but not on m or i, that is a (m/2’, r,~,r,~,2-4”)-stochastic-halver even i f each of its comparators is faulty with some fixed probability smaller than 112.

(F+ y ) ,

r ( X ) = { y l ( x ,y ) E E for some 5 E X } Definition: G = ( A , B , E ) is an (o,P,m,d)-expander if ]AI = IBI = m, the degree of every vertex in G is d, and for every set of vertices X , s.t. 5 om,

1x1

Ir(x)l2 PIXI. Theorem 2 For any o < 1 and p such that oP < 1 there is an explicit construction of an ( a ,p, m, d)-expander with d 5

w.

Proof: The construction is similar to the halver component in [AKS83], only the analysis is different, taking into account the faulty comparators. Let X = { X I , ..., X t } and Y = {K, ...,yZ} denote two sets of l = m/2‘ registers each. The input is given in the 2 l registers X U Y. The lower output set is Y and the upper output set is X. We con__ nect the registers in X to the registers of ? by an (77, d)-expander. Each edge of the expander corresponds to an ( &2-2it5 /“’)enforced-comparison between X E X to Y E Y moving the larger value t o Y and the smaller value to X . To schedule the comparisons we color the edges of the expander with d colors. The comparisons are executed according to the color order, thus, each register is involved in no more than one comparison at a time. The depth and width of each enforced comparator is by theorem 1 bounded by 2( L3 -)6lp/ ( l o g 2 3 / 1 8 ) ( T2 +log$$). The con-

w,

Proof: The construction in [LPS86] gives a d-regular bi-partite graph, with second largest eigenvalue bounded by 2 a , for any d = p 1, p prime. By [Alo86], if the second eigenvalue is bounded by 2& the expansion of each set of size bounded by om is at least P 2 -‘ d Thus, d = 4P(1-a) is sufficient 1-*p for expansion p. To guarantee the existence of d of the form p 1 , p prime, we take twice this value. 0

+

+

x

Definition: A set erepresents a value V , if at least (1 - c ) l X l registers in X store the value V .

e

218

If IH(X)l > q l , ( E ( X ) ( 5 q 2 [ / 4 , and IE(Y)I 5 q21/4, then the total number of registers that at the end of the execution of this component store elements that are larger than V is at least

struction contains d phases, each phase executes m/2' enforced-comparators in parallel. Thus, we get a network of depth c1 log and width c2m/2', where c1 and c2 depend on q and p , but not on m or i .

(5 + y),

Let E ( X ) and E ( Y ) denote the sets of registers in X and Y respectively, that are adjacent to faulty enforced-comparators. The probability that more than v 2 e / 4 out of the de enforced-comparators are faulty is bounded by

rle+

But the input set had no more than 202e values larger than V , and q21/4 faulty enforced-comparators can add no more than another q2e/4 values larger than V . Thus, IH(X)l must be smaller than qe. By a similar argument, Y has no more than values that are smaller than V . 0

5 2-4"

The fault tolerant network uses m = log N registers to simulate each register of the original sorting network. The invariant kept by the fault tolerant construction is that at least (1 - r)m of the m registers always store the correct value of the original network. The following construction enables the network to enforce the majority among the m registers in constant depth.

Thus, with probability 1- T4",E ( X ) and E ( Y ) are not larger than q21/4. A faulty enforced-comparator may output one of the input values in its two outputs, thus, can change the total number of values that are larger or smaller than V . The input set has no more than 2011 values smaller than V , and no more than 282l values larger than V . Thus, if there are no more than q21/4 faulty enforced-comparators, the upper output set has no more than ( 2 4 q2/4)! 5 (2d1 q ) e values smaller than V , and the lower output set has no more (282 q 2 / 4 ) l _< (202 + q ) l values larger than V .

+

1- q + q 2 / 2 q2 q2 - T ) e - -e q - q 2 / 4 (q 4

Definition: An (6, E , m,$)-majority expander is a construction with m inputs and m outputs. If more than ( 1 - b ) m of the inputs have the same value, then with probability 1 -$, this value ends up in at least (1- ~ ) m of the output registers. (I.e. if the input set &represents a values V , then with probability 1 - @ the output set €-represents V . )

+

+

Let H ( X ) denote the registers in X storing, at the end of the execution of this construction, values that are larger than V . If X # E ( X ) then the value of X does not increase while executing this construction, similarly the value of Y # E ( Y ) does not decrease. If X E H ( X ) - E ( X ) and Y # E ( Y ) is a neighbor of X then the value of Y at the end of the execution must be larger than the value of X , thus must be larger than V .

Theorem 4 For every fixed E > 0 there exists a construction of width O(m) and depth O(1 ) that is an (1/5, e, m, 2-4"+')-majorityexpander, even if each of its comparators is faulty with some fixed probability smaller than 1/2.

279

Proof: Let X = { X I , ...,-X },

denote the

even if each of the comparators in the construction is faulty with some fixed probability smaller than 1 / 2 .

m input registers, and let Y = { G ,..., Y,} denote an additional set of m registers. Let V denote the value that appears in more than 4 m / 5 inputs. The majority-expander netProof: (See figure 4 . ) Let X = work has four stages (see figure 2 ) . Stage 1 { X I , ...,X,} and Y = ...,Y,} be the two copies the values of 8 (the input values) t o Y. sets of input registers, let X be the upper outAt the end of stage 1, no more than 2 m / 5 regput set, and Y the lower output set. We use isters in X U Y store values smaller or larger 2-67- enforced- compar ators to compare each than V . In stage 2, the set X is connected Xe to I$, l = 1 , ..,m, moving the larger valto Y by an ( m ,~ / 4 , 62-4”)-stochastic-halver, , ues t o the Y registers and the smaller values where 6 = m i n [ e / 4 , 1 / 1 0 ] . All but ~ m / 4of to the X registers. the m largest values are moved to Y ,and at A register X i might have an incorrect value the end of stage 2, X has no more than ~ m / 4 after the comparison as a result of three reavalues larger than V and no more than m / 2 sons: (a) Xe had an incorrect value before the values smaller than V . Stage 3 copies again comparison. (b) yZ had an incorrect value the values of X to Y (erasing the previous (c) The comparison before the comparison. values of Y),at the end of this stage X U Y between Xe and I$ was faulty. has no more than 2 ~ m / 4values larger than V and no more than m values smaller than V . In The probability that more than m / 1 5 out stage 4 , Y is connected to with by another of the m comparisons are faulty is bounded ( m ,c / 4 , 6 , 2-4m)-stochastic-halver. After the execution of this stage, X ends up with no more than 3em/4 values larger than V , and no more than e m / 4 values smaller than V . Thus, the output X has at least ( 1 - ~ ) m Thus, with probability 1 - T4“at the end copies of the majority value V with probabilof this stage both X and Y 3/15-represent ity 1 - 2-4m+l. the correct values. Using the bounded depth ( 1 / 5 , 1 / 1 5 ,m, 2-4m+1)-majority-expander of Using the enforced-comparator and the theorem 4 we reduce the number of incorrect majority-expander we can now construct the values in each of these sets to m / 1 5 . 0 fault-tolerant comparator that simulates the perfect comparators of the original network. The following construction is used in the last phase of the network, where one correct Definition: An (6,m,p)-fauZt-tolerant value is extracted from m = log N registers, comparator is a construction with two sets among which at least (1 - ~ ) of m the values of m inputs, and two sets of m outputs. are correct. If the two input sets erepresent the values V1 and Vz respectively, then, with probabilDefinition: An (e, E,@)-majoritypreserver ity 1 - j, the upper output set e-represents is a network with l inputs and l / 2 outputs. If min[V,,Vz] and the lower output set eat least ( 1 - c ) l of the input values are equal represents max [VI ,V 2 ] . then this value appears in at least ( 1 - ~ ) l / 2 of the output registers.

{xL

x

Theorem 5 There exists a construction of depth O(1) and width O(m) that is a ( 1 / 1 5 ,m, 2 ~ 4 m ~ 3 ) - f a u Z t - t o Z e rcomparator nt

Theorem 6 For every E < 1 / 5 , and 0 5 i 5 logm there is an explicit construction

280

of a network of depth O(2') and width O ( m ) that is an (m/2', e , 2-3")-majority-pmserver, even i f each comparator in the construction is faulty with some fixed probability smaller than 1/2.

responding level of the original sorting network. In particular, at the end of this part of the fault tolerant sorting network, each set X ,1/15-represents the value that register X outputs in the original network.

The second part of the network reduces Proof: Let k' = m/2', X = {xl,...,x!/~) each set X;having at least (1 - 1/15)m corand Y = { K , ...,Y+}. Assume that the k' inrect copies to one register X ; storing one put registers are X U Y , and the l / 2 output correct value. We extract the correct copy registers are X. Denote by V the majority by log log N stages of majority-preservers value that appears in at least (1 - E)! input (see figure 5 ) The input to stage i, i = registers. The construction has three stages 0, ...,log log N - 1, is a set of m/2' registers (see figure 3). Stage 1 connects X to Y by an with at least (1 - 1/5)m/2' correct values. (m/2', c/4, c/4, 2-4")-~toch~tic-ha1~er, movStage i uses an (m/2', 1/5, 2-3m)-majoritying the smaller values to Y.Since X U have preserver. The depth of the majorityno more than ~ k 'values smaller than V , X has preserver is 0(2;), its width is O(m),and no more than d / 8 values smaller than V at with probability 1- T3"it produces a set of the end of this stage. Stage 2 copies the valm/2'+' registers with at least (1- 1/5)m/2'+' ues of X to Y . At the end of stage 2, X U Y registers storing the correct values. After has no more than 2 d / 8 values smaller than log m = log log N non-faulty iterations we are V , and no more than (2 1 / 4 ) d values larger left with one register storing the correct value. than V . In stage 3, Y is connected to X by an The depth of the second part of the network (m/2;, c/4, c/4, 2-4")-~tochastic-ha1~er,movis bounded by ing the smaller values to X . At the end of this stage, X has no more than 3 d / 8 values smaller than V and no more than d / 8 values larger than V . 0 = O(1og N). 0 rZN2') i=l

+

2.3.

The Complete Network

The width of the second part is O(mN) = O(N1og N), and with probability 1 N (log log N)2-3 log it produces the N correct outputs. Since the depth of any network that sorts N elements is at least logN, we proved:

The first part of the fault tolerant network simulates the comparators of the original sorting network. Let X I , ...,X N denote the N input registers of the original network. The first log log N levels generate m = log N copies of each input register. Using the log N copies of each original register we can simulate the comparators of the original network by (1/15, m, 2-4m+3 )-fault-tolerant comparators. The original sorting network has no more than N2 comparators, thus, with probability 1 - 2-2m+3, the outputs of every fault-tolerant comparator 1/15-represents the value of the corresponding register at the cor-

Theorem 7 Given a sorting network with depth d and width N, there is a fault tolerant sorting network of depth O ( d ) and width O(N log N ) that with probability 1- 1/N sorts correctly every list of N elements, even if each comparison in the network is faulty with some fixed probability smaller than 1/2.

281

3.

Remarks and Problems

Open

pages 307-314, Montvale, NJ, June 1968. [LPS86] A. Lubotzky, R. Phillips, and P. Sarnak. Ramanugan conjecture and explicit construction of expanders. In 18th Annual Symposium on Theory of Computing, pages 240-246, Berkeley, California, May 1986.

As mentioned before, our technique is applicable to any comparator based network. Thus, we can construct fault tolerant networks of depth O(1ogN) and width O ( N log N ) for merging two sorted lists of size N each, or for finding the maximum of N elements. The failure probability in both cases is smaller than 1/N, when each comparator is faulty with some fixed probability smaller than 1/2, independent of the other comparators.

[MS56] E. F. More and C. Shannon. Reliable circuits using less reliable relays i-ii. J. Franklin Institute, 262:191-208, 1956. Macwilliams [MS77] F.J. and N.J.A. Sloane. The Theory of Error-Cormcting Codes. NorthHolland, 1977.

A comparator network for finding the maximum of N elements uses O ( N ) comparators. Thus, our fault tolerant version of that network uses O ( N log N ) comparators. This bound matches the R(N1ogN) lower bound proven by Yao and Yao [YY85] for even a weaker model of faults. The optimality of the fault tolerant sorting network, and the fault tolerant merging network, in terms of width and total number of comparators, is still open. We showed in the introduction that some redundancy is essential for small failure probability but we do not know if R(1og N ) redundancy is necessary.

[Pip851 N. Pippenger. On networks of noisy gates. In 26th Annual Symposium on Foundations of Computer Science, pages 30-38, Portland, Oregon, October 1985. [Rud85] L. Rudolph. A robust sorting network. IEEE Transaction on Computers, C-34:326-335, 1985. [SS881

References [AKS83] M. Ajtai, J. Komlos, and E. Szemeredi. An o(n log n ) sorting network. Combinatorica, 3:l-19, 1983.

M. Schimmer and C. Stark. A correction network for N-sorter. In VLSI Algoritym and Architecture. 3rd Aegean Workshop on Computing, AWOC, pages 444-455, Corfu, Greece, June 1988.

[VN56] J. Von Neumann. Probabilistic logics and the synthesis of reliable organisms from unreliable components. In Automata Studies, pages 43-98, Princeton University Press, Princeton, NJ, 1956.

Eigenvalues and ex[A10861 N. Alon. panders. Combinatorica, 6:83-96, 1986.

[YY85] A. C. Yao and F. F. Yao. On fault-tolerant networks for sorting. SIAMJC, 14(1):120-128, 1985.

[Bat681 E. B. Batcher. Sorting networks and their applications. In Proc. AFIPS Spring Joint Computer Conference,

282

-

I

......

Figurc 1

- A p;+~-cnforccdcomparator madc up of pi-cdorccd

Figure 3 - Majorily Prcscrvcr.

283

comparators.

Majority Expuwlcr

EnforcedComparators

Majority ErpikJcr

Figurc 4 - A I5ull-Tolcrant Coniparalor which simulalcs a siniplc con~parison,

--

Majority

........

Prcscrvcr

~

~.

-

mR rcgistcrs

ni=logN rcgisten

-

F i g m 5 Extracting onc corrcct valuc from a sct that c-rcprcscnrs it.

284