finite permutation groups and finite simple groups - CiteSeerX

FINITE PERMUTATION GROUPS AND FINITE SIMPLE GROUPS PETER J. CAMERON 1. Introduction In the past two decades, there have been far-reaching developments in the problem of determining all finite non-abelian simple groups—so much so, that many people now believe that the solution to the problem is imminent. And now, as I correct these proofs in October 1980, the solution has just been announced. Of course, the solution will have a considerable effect on many related areas, both within group theory and outside. The purpose of this article is to consider the theory of finite permutation groups with the assumption that the finite simple groups are known, and to examine questions such as: which problems are solved or solvable under this assumption, and what important problems remain? Let us begin with an example. The best-known problem in finite permutation group theory is that of deciding whether there are any 6-transitive groups other than the symmetric and alternating groups. It was conjectured by Schreier that the outer automorphism group of a finite simple group is soluble. Such a conjecture is easily checked if the list of simple groups is known. (Indeed, at present it seems very likely that Schreier's conjecture will be proved in this way rather than directly.) Wielandt [67] reduced the first of these problems to the second: his result, refined by Nagao [46] and O'Nan [48], asserts that a 6-transitive group must be symmetric or alternating, provided that the composition factors of its proper subgroups have soluble outer automorphism groups. However, unless a direct proof of Schreier's conjecture can be found, this is the wrong way to settle the question. As we shall see, the determination of the finite simple groups enables us, with more effort, to determine all the 2-transitive groups. By inspection, none, except the symmetric and alternating groups, is 6-transitive. In order to discuss the consequences of knowing the finite simple groups, we must say something about the sense in which they are known. The multiplication tables of the finite simple groups are "recursive", in the sense that if they are encoded by Godel numbers in some way, the resulting set M of numbers is recursive. This means that we could in principle construct a machine which would decide, given a natural number n, whether or not n e M. By the results of Davis, Matijasevic and Robinson on Hilbert's tenth problem (see [18]), we know that M can even be expressed as the set of positive values of a polynomial. However, even if such a polynomial were explicitly known, it would not constitute a satisfactory solution to the classification problem. Each group must be known sufficiently well that questions about its automorphisms, permutation representations, local subgroups, and so on, can be answered. In this article, we will often invoke the following hypothesis (S). Its use could be compared to that of the continuum hypothesis or the Riemann hypothesis in other parts of mathematics (though the analogy does not run very deep). Received 24 April, 1980. [BULL. LONDON MATH. S O C , 13 (1981),

1-22]

2

PETER J. CAMERON

(S) A non-abelian finite simple group is an alternating group, a group of Lie type, or one of finitely many sporadic groups. The alternating groups require little comment: there is one such simple group for each degree n ^ 5. Groups of Lie type are the Chevalley groups [15] and twisted analogues discovered by Steinberg [60], Suzuki [61], Ree [53, 54] and Tits [63]. It will be convenient to divide them into two types: the classical linear, symplectic, unitary and orthogonal groups (which fall into six families, each parametrised by a dimension n and a field order q), and the exceptional groups (comprising ten families each parametrised by a field order). Sometimes, we assume a more refined version of (S) in which the only sporadic groups are those whose existence is known or suspected at present (see [37] for the list), and these are "known" in the sense previously described; it will be clear how to modify the arguments should finitely many more groups be discovered. Obviouly our knowledge of these groups is a function of time. Examples in the paper illustrate how the present knowledge is used. It should be stressed that the most important problem to be faced now is that of obtaining more information about these groups, especially those of Lie type. To make the paper self-contained, Sections 2 and 3 outline the elementary theory of permutation groups, explaining the central role played by the primitive groups. The next section discusses a theorem stated by Michael O'Nan and Leonard Scott at the symposium on Finite Simple Groups at Santa Cruz in the summer of 1979, concerning the socle of a primitive group. (Although the proof of this theorem is elementary, and all or part of it may have been known previously to some people, I know of no explicit reference in the literature. The proceedings of the Santa Cruz symposium are to appear in the Proceedings of Symposia in Pure Mathematics, published by the American Mathematical Society; but the theorem quoted here is taken from the preliminary version circulated to participants. I would like to thank those people who made their copies available to me.) The O'Nan-Scott theorem enables many questions about primitive groups G to be reduced to the case where T ^ G ^ Aut (T) for some non-abelian simple group T; this is where hypothesis (S) comes into play. The next four sections discuss the implications of (S) for some classical problems: the determination of primitive groups of small rank and, in particular, all doubly transitive groups; the orders and degrees of primitive groups not containing the alternating group; and remarks on the problem of finding all primitive groups. Section 9 is more in the spirit of traditional permutation group theory, dealing with problems which are not solved or trivialised by (S). Section 10 discusses briefly some issues raised by the problem of computer-aided recognition of permutation groups; it is based on discussions with John Cannon. The final section considers related areas, especially automorphism groups of combinatorial structures and infinite permutation groups. The proof of (S), when it is completely checked, will be an achievement of great magnitude, involving thousands of pages of group theory written by an army of researchers. However, it raises philosophical problems of a very different kind from those posed by a recent well-publicised breakthrough, the proof of the fourcolour conjecture by Appel and Haken [1]: human beings are less reliable than properly-programmed machines. It is certain that the first published proof of (S), spread over many journals'and theses, will contain mistakes. Thus, it is important

FINITE PERMUTATION GROUPS AND FINITE SIMPLE GROUPS

3

that the work of revision and the search for better proofs should continue. The same applies to the material of Sections 5-7 of this paper, where it is also desirable for aesthetic reasons that proofs should be found which do not depend on (S). Nowhere is this more so than in the classification of doubly transitive groups. The material in this paper grew out of a discussion of the aftermath of the Santa Cruz symposium; those who attended the meeting will recognise my debt to Michael O'Nan and Leonard Scott. I would also like to record my gratitude to the University of Sydney, for giving me the opportunity to deliver lectures in which the ideas here presented were worked out; to the audience at those lectures, for their help in the discussions; to John Cannon, for introducing me to the material in Section 10; and to Peter Neumann, who suggested a large number of improvements to the paper.

2. Transitivity and primitivity A permutation group on a set ft is simply a subgroup of the symmetric group on Q. It is convenient, however, to have a more general concept, an action of G on ft, which may be regarded as a homomorphism from G into the symmetric group. (Of course the image, which we write as Gn, is a permutation group, and can be identified with G if the action is faithful.) The image of a e ft under the permutation corresponding to g e G will be written ag. We say that G acts transitively on ft if, for any a, /? e ft, there exists g e G with ag = p. In general, the relation ~ on ft defined by the rule that a ~ P if ocg = P for some g e G is an equivalence relation, whose equivalence classes are called orbits; G acts transitively on each orbit. We call the group G* of permutations induced on an orbit O by G a transitive constituent of G. Thus, the study of arbitrary permutation groups "reduces" to that of transitive groups. (However, it should be noted that G is not determined by its transitive constituents. It is contained in their cartesian product, and is in fact a subcartesian product: that is, the projection map associated with each factor maps G onto that factor; but more cannot be said in general. Subsequent "reductions" will involve a similar loss of information.) If H is a subgroup of G, there is a transitive action of G on the set (G : H) of right cosets of H, by right multiplication: (Hx)g = H(xg). Let ft be a set on which G acts transitively, and Ga = {g e G | (G : GJ satisfying (/?)# = (Pg) for all P e ft, g € G. (It is defined by P = {g e G | ag = /?}.) Thus the study of transitive groups is equivalent to that of coset spaces. A permutation group G is called semiregular if Ga = 1 for all a e ft, and regular if it is transitive and semiregular. For regular groups G, the bijection identifies ft with G, giving the right regular representation of G. It can be shown that a permutation group is transitive if and only if its centraliser in the symmetric group is semiregular, and vice versa (Wielandt [68], page 9). In particular, the centraliser of a regular group is regular. Specifically, the centraliser of the right regular representation is the left regular representation, in which a group element g acts as left multiplication by g~x. Suppose G has a normal subgroup N which is regular on ft. Then, as above, we may identify ft with N. If a eft is mapped to the identity element of N, then the actions of Ga on ft and on N by conjugation agree under this identification: that is, for fi e ft, g e Ga, (Pg)4> = g~x(p(f))g (Wielandt [68], page 27).

4

PETER J. CAMERON

Let G act on Q. Then G is transitive if the only G-invariant subsets of Q are trivial (Q and the empty set). Consider now the related assertion that the only G-invariant partitions of Q are trivial (the partition into singletons and the partition with a single part). Obviously this condition implies that G is transitive provided that |Q| > 2. A transitive group with no nontrivial invariant partitions is called primitive. Also, a nontrivial G-invariant partition is called a system of imprimitivity, and its parts are blocks (of imprimitivity). A non-identity normal subgroup of a primitive group is transitive (Wielandt [68], page 17)—indeed, if N is normal in G, then the partition of Q, into orbits of N is invariant under G. Primitivity of the action on a coset space can be recognised group-theoretically: the action of G on (G : H) is primitive if and only if if is a maximal subgroup of G (Wielandt [68], page 15). For a natural number fc < |Q|, a permutation group G on Q is k-transitive if its induced action on the set of ordered /c-tuples of distinct elements of Q is transitive. The symmetric group of degree n is n-transitive, and the alternating group is (n — 2)transitive. Apart from these examples, no known finite group is 6-transitive, and only two (the Mathieu groups M 1 2 and M 2 4 ) are 5-transitive. Note that a 2-transitive group is primitive (Wielandt [68], page 20).

3. Wreath products Let C be a group, and D a permutation group on a set A. The wreath product C Wr D is the split extension of a base group CA (the cartesian product of |A| copies of C) by D, where D acts on CA by permuting the factors as it does the elements of A. Identifying CA with the set of functions / : A -> C, we have d~ifd(d) = f(Sd~i) for deD,de&. Suppose C acts on a set F. Then there are two natural actions of C WrD. 1. The imprimitive action on F x A. The base group acts on the first coordinate, by the rule (y, d)f = (y/( 1: the partition F x A = [J F x {3} is a system of imprimitivity. It is universal in the following sense. Se D

PROPOSITION 3.1. Let G act imprimitively on fi. Let F be a block of imprimitivity, A an index set for the parts of the corresponding system of imprimitivity, C = Gp (where G r denotes the setwise stabiliser of F), and D = GA. Then Q can be identified with F x A in such a way that Gn is a subgroup of C Wr D (in its imprimitive representation).

Thus, if G is imprimitive, it is built from smaller permutation groups. If Q is finite, we may continue refining the decomposition to obtain primitive components for G. However, there is no Jordan-Holder theorem here, since different refinements may yield different primitive components. (Consider, for example, the right regular representation of the symmetric group S 3 . If we choose for F the subgroup of order 3, then we have C = C 3 , D = C2, where Cn is the cyclic group of order n in its regular representation; whereas if F is a subgroup of order 2, then C = C2 but D = S 3 in its natural action on 3 letters.) It is also true that the primitive components do not determine the group uniquely; the comment concerning the reduction of arbitrary groups to transitive ones applies here too.


5

2. The product action on FA, where the base group acts coordinatewise (that is, if 4> e F A , / e CA, then (/)({dd~x) for $ 6 TA, d e D). This action is usually primitive: PROPOSITION 3.2. Suppose D is transitive on A, and C is primitive but not regular on F. Then C WrZ), in its product action on FA, is primitive.

Note that a primitive regular group is necessarily cyclic of prime order (Wielandt [68], page 17), since the trivial subgroup of such a group is maximal. If |A| > 1, the hypotheses are also necessary for primitivity.

4. A theorem of O'Nan and Scott Let G be a primitive permutation group on a finite set Q. Suppose that N is a minimal normal subgroup of G. Then N is transitive. Now the centraliser CG(N) is also a normal subgroup. If CC{N) ^ 1, then CC{N) is transitive, whence N and CG{N) are both regular; they are equal or distinct according as N is abelian or not. In this case N and CC(N) are minimal normal subgroups, and G has no further minimal normal subgroups (since such subgroups centralise each other). Moreover, N and CG{N) are isomorphic, since they are right and left regular representations of the same group. If, on the other hand, CG(N) = 1, then N is the unique minimal normal subgroup of G, and G is isomorphic to a subgroup of Aut(iV), the automorphism group of N. (For, in any group G with normal subgroup N, the action of G on N by conjugation provides a homomorphism from G into Aut (NJ with kernel CG(N).) In either case, we see that the socle of G (the product of its minimal normal subgroups) is a direct product of isomorphic simple groups. Let S,S!,..., Sh be groups, and for 1 ^ i ^ h let 0 , : S -• S, be an isomorphism. The diagonal subgroup of Sx x...xSh (relative to the given isomorphisms) is the subgroup D = {(scf) t ,..., s(f)h) \ s e S}. THEOREM 4.1 (O'Nan-Scott). Let G be a primitive permutation group on Q, with degree n and socle N. Then one of the following occurs.

(i) N is elementary abelian of order pd and regular, n = pd, where p is prime and d ^ 1. (ii) N = Tx x ...xTm, where Tl,...,Tm group T. Moreover, either

are all isomorphic to a fixed simple

(a) T is the socle of a primitive group Go of degree n0, and G ^ G 0 Wr5 m (with the product action), where n = n%; or (b) N n Ga = D, x... x Dh where m = kl for some k, Dt is the diagonal subgroup of T(i_X)k+{ x... x Tik, and n = \T\{k~1)l. In view of the importance of this theorem for the next four sections, a proof will be outlined. If N is abelian then its simple factors are cyclic of prime order, and N is elementary abelian; so (i) holds. Otherwise, (ii) follows from the reasoning preceding the theorem. Since Ga is a maximal subgroup of G, it follows that N n Ga is a maximal Ga-invariant subgroup of N. If N n Ga projects onto one factor of the direct product, then it projects onto all, and it is a product of diagonal subgroups; it can be

6

PETER J. CAMERON

verified that (ii)(b) holds. Otherwise, the projection St of N n Ga into Tx is a maximal (NG{TX) n Ga)-invariant subgroup of Tl; its projection S, into 7] is the image of Sj under an isomorphism from Tx to 7> and N n Ga is the direct product of the projections Sx,..., Sm. (For, if R{ is a ( ^ ( T J n Ga)-invariant subgroup of Tu and Ri the corresponding subgroup of 7], then R{x ...x Rm is a Ga-invariant subgroup of N.) Thus (ii)(a) holds. Remarks. 1. In case (i), N is the additive group of a vector space V of dimension d over the field GF(p). Identifying Q with V, we see that Ga is a group of automorphisms of N, that is, a subgroup of the general linear group GL (V) = GL (d, p), Moreover, Ga is irreducible, since any invariant subspace would be a block of imprimitivity. Conversely, if H is an irreducible linear group on a finite vector space V, then the split extension G = VH is a primitive group falling under case (i) of the theorem. 2. In case (ii)(b), the permutation group induced by G on {r i 5 ..., 7^,,} has {7 l 5 ..., 7^} as a block of imprimitivity. The group induced on the set of blocks is transitive. Also, the stabiliser of a block leaves invariant no nontrivial partition of that block. If it is the trivial group (with k = 2) then G has two minimal normal subgroups; otherwise, it is primitive, and N is the unique minimal normal subgroup of G. 3. A group of type (ii)(b) with / > 1 is a subgroup of Go Wr S, (with the product action), where Go is primitive of type (ii)(b) with / = 1. Thus the study of primitive groups is "reduced" to the following four parts: (i) irreducible linear groups over finite prime fields; (ii) the product action of wreath products; (iii) groups of type (ii)(b) with / = 1; (iv) primitive groups with nonabelian simple socle. (Of course, the earlier remarks about "reductions" apply here.) A complete classification of irreducible linear groups appears somewhat remote, and may not be very helpful if it were known. So we often ignore this case, merely remarking that for some problems (examples of which are given below) it presents no difficulties. In a similar way, the second and third of the above topics can often be handled easily in a particular case. The crucial point at which our hypothesis (S) needs to be used is in the fourth problem, the determination of primitive groups with simple socle. This is equivalent to determining all maximal subgroups of groups G satisfying T ^ G ^ Aut(T), where T is a nonabelian simple group. (This question asks for more than a determination of the maximal subgroups of T, since there can be maximal subgroups H of G for which H n T is not maximal in T.)

5. Rank and multiple transitivity The rank of a transitive permutation group G on fi is the number of orbits of G in its action on Q x Q . (This is equal to the number of orbits of Ga on Q, for a e Q,


7

(see Section 9).) Thus, for |Q| > 1, G has rank at least 2, with equality if and only if it is 2-transitive. (The diagonal {(a, a) | a e 0 } is one G-orbit.) Is there any relation between the rank of a primitive group and the structure of its socle? Clearly no such relation is possible in case (i) of Theorem 4.1, since the split extension of V by GL (V) is 2-transitive for any finite vector space V. In case (ii), however, we have the following result. PROPOSITION 5.1. Let G be a primitive group of rank r on Q, whose socle is a direct product ofm nonabelian simple groups. Then m ^ r - 1 . Moreover, assuming (S), in case (ii)(b) of Theorem 4.1, the order of each simple group (and hence the degree and order of G) is bounded by a function ofr.

Proof. It is not difficult to show that, if Go is a transitive group of rank r0, then the rank of G 0 WrS m (in its product action) is r' = I °

I; and of course, if

G ^ Go Wr Sm, then r ^ r'. Thus, in case (ii)(a), given r, both m and r0 are bounded, and m ^ r — 1. Let h denote the number of orbits of the group K of permutations of T generated by automorphisms and inversion (the map t\—• t" 1 ). Then a group of type (ii)(b) with / = 1 has rank r satisfying r ^ l + (/i —l)Qm] (see below). So, given r, both h and m are bounded. Since T contains elements of at least four different orders, we have h ^ 4 and m < r — l. The largest subgroup of Sn in which Tx x . . . x Tm is normal, in case (ii)(b) with / = 1, is an extension of Ti x . . . x Tm by Out[T)xS m , where Out(T) is the outer automorphism group of T; the automorphisms act in the same way on each factor, and the symmetric group permutes the factors. The subgroup T 1 x . . . x T m _ 1 is regular, though not normal, so Q can be identified with Tm~x. Now the stabiliser of the identity is Aut {T)xSm, and is generated by elements of three types: (i) automorphisms of T, acting in the same way on each coordinate; (ii) permutations of the coordinates (elements of S m _ x ); and (iii) the m a p

(tl,...,tm-l)\-+(t;l,tilt2,...,tiltm_l)

(induced by the transposition (lm) e Sm). Thus, as t runs through a set of representatives of the nontrivial K-orbits, the elements (t,t, ...,t,l,...,i) (with i entries t, for 1 ^ i ^ [^m]), lie in different suborbits. This gives the bound r ^ l + (h — \)[jm~\. It can be attained if m = 2, but it is very far from best possible for m > 2. For example, when T = A5, we have h = 4, but the group (A 5 ) 3 (C 2 x S3) of degree 3600 has rank 17. To prove the last assertion in the proposition, it is necessary to show that the order of the simple group T is bounded by a function of h. This is true for the known simple groups. I do not know a proof independent of (S). Clearly we may replace h by the number hx of orbits of Aut (T) on T. The question is related to the well-known fact that the order of any finite group is bounded by a function of the number of its conjugacy classes (the orbits of the group of inner automorphisms). See Burnside [9], page 461. As a corollary of Proposition 5.1, we obtain the following result of Burnside ([9], page 202); more properly, Proposition 5.1 should be seen as a generalisation of Burnside's theorem.

8

PETER J. CAMERON

PROPOSITION 5.2. A 2-transitive group has a unique minimal normal subgroup, which is elementary abelian or simple.

From this and the work of Huppert, Hering, Maillet, Howlett, Curtis, Kantor, Seitz, and others, comes a major consequence of (S). THEOREM 5.3 (S).

All finite 2-transitive groups are known.

Proof. Suppose first that the minimal normal subgroup N of such a group G is elementary abelian. Then Ga is a subgroup of GL (d, p), transitive on nonzero vectors of the vector space. The soluble 2-transitive groups were determined by Huppert [36], so we may assume that Ga is insoluble. Hering [26] showed that Ga has a unique nonabelian composition factor, and subsequently [28] he examined the known simple groups and determined all such situations in which each could occur as such a composition factor. In the other case (when N is simple), we have G ^ Aut (N). If N is an alternating group Ak, then G = Ak or Sk (for k ^ 5, k ± 6); all the 2-transitive representations of these groups were determined by Maillet [43]. When N is a group of Lie type, the corresponding problem was solved by Curtis, Kantor and Seitz [17] (see also Howlett [35]). It is implicit in (S) that the sporadic groups can be handled by ad hoc arguments.

We illustrate one method of proving theorems of this type, by dealing with the symmetric groups. Suppose G = Sk, and G acts 2-transitively on Q. There is a conjugacy class C of ^/c(/c — 1) elements of order 2 in G (the transpositions). Choose a e Q. For each further point P of Q, there is a member of C which interchanges a and p. If H = Ga, then \Sk: H\ = |Q| < 1 + |C| = |(/c 2 -k + 2). Comparing this with known lower bounds for the index of subgroups of Sk (see the next section), we conclude that H = Sk.1 with a few known exceptions. We list here the simple groups T which can occur as minimal normal subgroups of 2-transitive groups of degree n, according to Theorem 5.3. The number k is the maximum degree of transitivity of a group G with socle T. T

n

k

Remarks

An,n^5 PSL(d, q), d 2 2

n ( ^ _ ! ) / ( , _ i)

PSU(3,g) 2 B2(q) (Suzuki) 2 G2{q)(Ree) PSp(22 q = 22a+1>2 q = 32a + 1 > 3 d>2 d>2 Two representations

2

2 2 2 4 3 5 3 4 5 2 2

Two representations Two representations

Two representations


9

Notes. 1. There is one duplication in the table, explained by the isomorphism between A5 and PSL(2,4). The additional isomorphisms PSL(2,5) = A5, PSL(2,7)^PSL(3,2), PSL(2,9) s A6, and PSL(4,2)^X 8 give examples of abstract groups with more than one 2-transitive permutation representation, in addition to the examples noted in the remarks and the cases PSL(2, 8), PSL(2,11), PSp{2d,2)(d > 2),A1 and M n . 2. The socle T is /c-transitive in all cases except T = An ((n — 2)-transitive), T = PSL(2, q), q odd (2-transitive), and T = PSL(2, 8), degree 28 (1-transitive). 3. The groups PSL(2,2)^S 3 , PSL(2, 3) s X4, PSU(3,2) and 2B2{2) are soluble, with degrees 3, 4, 9 and 5 respectively; 2G2(3) has socle PSL(2, 8), of degree 28, as a subgroup of index 3; and PSp(4, 2) has socle of index 2 which acts as A6 of degree 6 and PSL(2,9) of degree 10 (so that PSp(4,2) s S6). 4. Apart from the fact that any simple group of degree n is contained in An, the inclusions among groups T in the table are as follows: PSL(2,11) (degree 11) ^ M n (degree 11); PSL(2, 11) (degree 12) ^ M 12 ; M u (degree 12) ^ M 12 ; A, (degree 15) < PSL(4, 2); PSL(2,23)^M 2 4 ; PSL(2, 8) (degree 28) ^ PSp(6, 2) (degree 28); PSU(3, 3) (degree 28) ^ PSp(6, 2) (degree 28). Note also that PSL(2, 7) is contained in the holomorph of C2, a 3-transitive group of degree 8 with elementary abelian socle. 5. Some of these representations correspond to familiar geometrical objects. For example, PSL(d, q) acts on the points and hyperplanes of the projective space PG(d — l,q); PSU(3,