Basic Concepts in Matrix Algebra. • An column array of p elements is called a
vector of dimension p and is written as xp×1. = . . . . x1 x2 ... xp. . . .
.
Basic Concepts in Matrix Algebra • An column array of p elements is called a vector of dimension p and is written as x1 x xp×1 = ..2 . . xp • The transpose of the column vector xp×1 is row vector
x0 = [x1 x2 . . . xp] • A vector can be represented in p-space as a directed line with components along the p axes. 38
Basic Matrix Concepts (cont’d) • Two vectors can be added if they have the same dimension. Addition is carried out elementwise.
x+y
=
x1 x2 ... xp
+
y1 y2 ... yp
=
x1 + y1 x2 + y2 ... xp + yp
• A vector can be contracted or expanded if multiplied by a constant c. Multiplication is also elementwise.
cx
= c
x1 x2 ... xp
=
cx1 cx2 ... cxp
39
Examples
x
=
2 h i 0 1 and x = 2 1 −4 −4
12 6×2 2 6×x=6× 1 = 6 6×1 = −24 6 × (−4) −4
x+y
=
2 5 2+5 7 1 + −2 = 1 − 2 = −1 −4 0 −4 + 0 −4 40
Basic Matrix Concepts (cont’d) • Multiplication by c > 0 does not change the direction of x. Direction is reversed if c < 0.
41
Basic Matrix Concepts (cont’d) • The length of a vector x is the Euclidean distance from the origin v u p uX x2 Lx = u t j j=1
• Multiplication of a vector x by a constant c changes the length: v v u p u p uX uX u 2 2 c xj = |c|u x2 Lcx = t t j = |c|Lx j=1 j=1
• If c = L−1 x , then cx is a vector of unit length. 42
Examples
The length of x
Lx =
= q
2 1 −4 −2
is
(2)2 + (1)2 + (−4)2 + (−2)2 =
√
25 = 5
Then
z
2
1 1 = × 5 −4 −2
0.4
0.2 = −0.8
−0.4
is a vector of unit length. 43
Angle Between Vectors • Consider two vectors x and y in two dimensions. If θ1 is the angle between x and the horizontal axis and θ2 > θ1 is the angle between y and the horizontal axis, then x y cos(θ1) = 1 cos(θ2) = 1 Lx Ly x y sin(θ1) = 2 sin(θ2) = 2 , Lx Ly If θ is the angle between x and y, then cos(θ) = cos(θ2 − θ1) = cos(θ2) cos(θ1) + sin(θ2) sin(θ1). Then x y + x2y2 cos(θ) = 1 1 . LxLy 44
Angle Between Vectors (cont’d)
45
Inner Product • The inner product between two vectors x and y is
x0y =
p X
xj yj .
j=1
• Then Lx =
√
x0x, Ly =
q
y0y and
cos(θ) = q
x0y (x0x)
q
(y 0 y )
• Since cos(θ) = 0 when x0y = 0 and cos(θ) = 0 for θ = 90 or θ = 270, then the vectors are perpendicular (orthogonal) when x0y = 0. 46
Linear Dependence • Two vectors, x and y, are linearly dependent if there exist two constants c1 and c2, not both zero, such that c1 x + c2 y = 0
• If two vectors are linearly dependent, then one can be written as a linear combination of the other. From above:
x = (c2/c1)y • k vectors, x1, x2, . . . , xk , are linearly dependent if there exist constants (c1, c2, ..., ck ) not all zero such that k X j=1
cj xj = 0 47
• Vectors of the same dimension that are not linearly dependent are said to be linearly independent
Linear Independence-example Let
1
x1 = 2 ,
1
x2 =
1 0 , −1
1
x3 = −2
1
Then c1x1 + c2x2 + c3x3 = 0 if c1 + c2 + c3 = 0 2c1 + 0 − 2c3 = 0 c1 − c2 + c3 = 0 The unique solution is c1 = c2 = c3 = 0, so the vectors are linearly independent. 48
Projections • The projection of x on y is defined by
x0y 1 x0y y. Projection of x on y = 0 y = yy Ly Ly • The length of the projection is |x0y| |x0y| Length of projection = = Lx = Lx| cos(θ)|, Ly LxLy where θ is the angle between x and y.
49
Matrices A matrix A is an array of elements aij with n rows and p columns: A=
a11 a12 · · · a1p a21 a22 · · · a2p ... ... ... ... an1 an2 · · · anp
The transpose A0 has p rows and n columns. column of A a11 a21 · · · a a22 · · · 0 A = 12 ... ... ... a1p a2p · · ·
The j-th row of A0 is the j-th an1 an2 ... anp
50
Matrix Algebra • Multiplication of A by a constant c is carried out element by element. cA =
ca11 ca12 · · · ca1p ca21 ca22 · · · ca2p ... ... ... ... can1 can2 · · · canp
51
Matrix Addition Two matrices An×p = {aij } and Bn×p = {bij } of the same dimensions can be added element by element. The resulting matrix is Cn×p = {cij } = {aij + bij }
C = A+B = =
a11 a12 · · · a1p a21 a22 · · · a2p ... ... ... ... an1 an2 · · · anp
+
b11 b12 · · · b1p b21 b22 · · · b2p ... ... ... ... bn1 bn2 · · · bnp
a11 + b11 a12 + b12 · · · a1p + b1p a21 + b21 a22 + b22 · · · a2p + b2p ... ... ... ... an1 + bn1 an2 + bn2 · · · anp + bnp
52
Examples
"
"
2 1 −4 5 7 0
6×
2 1 −4 5 7 0
"
#
2 −1 0 3
"
+
#0
=
#
"
=
2 1 5 7
2 5 1 7 −4 0
12 6 −24 30 42 0
#
"
=
4 0 5 10
#
#
53
Matrix Multiplication • Multiplication of two matrices An×p and Bm×q can be carried out only if the matrices are compatible for multiplication: – An×p × Bm×q : compatible if p = m. – Bm×q × An×p: compatible if q = n. The element in the i-th row and the j-th column of A × B is the inner product of the i-th row of A with the j-th column of B.
54
Multiplication Examples
"
2 0 1 5 1 3
"
"
2 1 5 3
#
" # 1 4 2 10 × −1 3 = 4 29 0 2
#
1 4 −1 3
"
×
#
1 4 −1 3
"
×
2 1 5 3
#
"
=
#
"
=
1 11 2 29
22 13 13 8
#
#
55
Identity Matrix • An identity matrix, denoted by I, is a square matrix with 1’s along the main diagonal and 0’s everywhere else. For example, "
I2×2 =
1 0 0 1
#
1 0 0 and I3×3 = 0 1 0 0 0 1
• If A is a square matrix, then AI = IA = A.
• In×nAn×p = An×p but An×pIn×n is not defined for p 6= n.
56
Symmetric Matrices • A square matrix is symmetric if A = A0.
• If a square matrix A has elements {aij }, then A is symmetric if aij = aji.
• Examples "
4 2 2 4
#
5 1 −3 1 12 −5 −3 −5 9
57
Inverse Matrix • Consider two square matrices Ak×k and Bk×k . If AB = BA = I then B is the inverse of A, denoted A−1.
• The inverse of A exists only if the columns of A are linearly independent. • If A = diag{aij } then A−1 = diag{1/aij }.
58
Inverse Matrix • For a 2 × 2 matrix A, the inverse is
A−1 =
"
a11 a12 a21 a22
#−1
1 = det(A)
"
a22 −a12 −a21 a11
#
,
where det(A) = (a11 × a22) − (a12 × a21) denotes the determinant of A.
59
Orthogonal Matrices • A square matrix Q is orthogonal if QQ0 = Q0Q = I, or Q0 = Q−1. • If Q is orthogonal, its rows and columns have unit length (q0j qj = 1) and are mutually perpendicular (q0j qk = 0 for any j 6= k).
60
Eigenvalues and Eigenvectors • A square matrix A has an eigenvalue λ with corresponding eigenvector z 6= 0 if Az = λz • The eigenvalues of A are the solution to |A − λI| = 0. • A normalized eigenvector (of unit length) is denoted by e. • A k × k matrix A has k pairs of eigenvalues and eigenvectors λ1, e1
λ2, e2
...
λk , ek
where e0iei = 1, e0iej = 0 and the eigenvectors are unique up to a change in sign unless two or more eigenvalues are equal. 61
Spectral Decomposition • Eigenvalues and eigenvectors will play an important role in this course. For example, principal components are based on the eigenvalues and eigenvectors of sample covariance matrices. • The spectral decomposition of a k × k symmetric matrix A is A = λ1e1e01 + λ2e2e02 + ... + λk ek e0k = [e1 e2 · · · ek ]
λ1 0 · · · 0 0 λ2 · · · 0 ... ... . . . ... 0 0 · · · λk
[e1 e2 · · ·
ek ]0
= P ΛP 0 62
Determinant and Trace • The trace of a k × k matrix A is the sum of the diagonal elements, i.e., P trace(A) = ki=1 aii • The trace of a square, symmetric matrix A is the sum of the eigenvalP P ues, i.e., trace(A) = ki=1 aii = ki=1 λi • The determinant of a square, symmetric matrix A is the product of the Q eigenvalues, i.e., |A| = ki=1 λi
63
Rank of a Matrix • The rank of a square matrix A is – The number of linearly independent rows – The number of linearly independent columns – The number of non-zero eigenvalues
• The inverse of a k × k matrix A exists, if and only if rank(A) = k i.e., there are no zero eigenvalues 64
Positive Definite Matrix • For a k × k symmetric matrix A and a vector x = [x1, x2, ..., xk ]0 the quantity x0Ax is called a quadratic form • Note that x0Ax =
Pk Pk i=1 j=1 aij xi xj
• If x0Ax ≥ 0 for any vector x, both A and the quadratic form are said to be non-negative definite. • If x0Ax > 0 for any vector x 6= 0, both A and the quadratic form are said to be positive definite. 65
Example 2.11 2−2 • Show that the matrix of the quadratic form 3x2 + 2x 1 2
√
2x1x2 is
positive definite.
• For "
A=
√
√3 − 2 − 2 2
#
,
the eigenvalues are λ1 = 4, λ2 = 1. Then A = 4e1e01 + e2e02. Write
x0Ax = 4x0e1e01x + x0e2e02x = 4y12 + y22 ≥ 0, and is zero only for y1 = y2 = 0. 66
Example 2.11 (cont’d) • y1, y2 cannot be zero because "
y1 y2
#
"
=
e01 e02
#"
x1 x2
#
0 = P2×2 x2×1
with P 0 orthonormal so that (P 0)−1 = P . Then x = P y and since x 6= 0 it follows that y 6= 0.
• Using the spectral decomposition, we can show that: – A is positive definite if all of its eigenvalues are positive. – A is non-negative definite if all of its eigenvalues are ≥ 0. 67
Distance and Quadratic Forms • For x = [x1, x2, ..., xp]0 and a p × p positive definite matrix A, d2 = x0Ax > 0 when x 6= 0. Thus, a positive definite quadratic form can be interpreted as a squared distance of x from the origin and vice versa.
• The squared distance from x to a fixed point µ is given by the quadratic form (x − µ)0A(x − µ).
68
Distance and Quadratic Forms (cont’d) • We can interpret distance in terms of eigenvalues and eigenvectors of A as well. Any point x at constant distance c from the origin satisfies
x0Ax = x0(
p X
λj ej e0j )x =
j=1
p X
λj (x0ej )2 = c2,
j=1
the expression for an ellipsoid in p dimensions. −1/2
• Note that the point x = cλ1 e1 is at a distance c (in the direction of e1) from the origin because it satisfies x0Ax = c2. The same is true −1/2 for points x = cλj ej , j = 1, ..., p. Thus, all points at distance c lie on an ellipsoid with axes in the directions of the eigenvectors and with −1/2 lengths proportional to λj . 69
Distance and Quadratic Forms (cont’d)
70
Square-Root Matrices • Spectral decomposition of a positive definite matrix A yields A=
p X
λj ej e0j = P ΛP,
j=1
with Λk×k = diag{λj }, all λj > 0, and Pk×k = [e1 e2 ... ep] an orthonormal matrix of eigenvectors. Then A−1 = P Λ−1P 0 = 1/2
• With Λ1/2 = diag{λj
p X 1 j=1 λj
ej e0j
}, a square-root matrix is
A1/2 = P Λ1/2P 0 =
p q X
λj ej e0j
j=1 71
Square-Root Matrices The square root of a positive definite matrix A has the following properties: 1. Symmetry: (A1/2)0 = A1/2 2. A1/2A1/2 = A 3. A−1/2 =
Pp −1/2 0 = λ e e j j j=1 j
P Lambda−1/2P 0
4. A1/2A−1/2 = A−1/2A1/2 = I 5. A−1/2A−1/2 = A−1 Note that there are other ways of defining the square root of a positive definite matrix: in the Cholesky decomposition A = LL0, with L a matrix of lower triangular form, L is also called a square root of A. 72
Random Vectors and Matrices • A random matrix (vector) is a matrix (vector) whose elements are random variables. • If Xn×p is a random matrix, the expected value of X is the n×p matrix E(X) =
E(X11) E(X12) E(X21) E(X22) ... ... E(Xn1) E(Xn2)
· · · E(X1p) · · · E(X2p) , . .. ··· · · · E(Xnp)
where E(Xij ) =
Z ∞ ∞
xij fij (xij )dxij
with fij (xij ) the density function of the continuous random variable Xij . If X is a discrete random variable, we compute its expectation as a sum rather than an integral. 73
Linear Combinations • The usual rules for expectations apply. If X and Y are two random matrices and A and B are two constant matrices of the appropriate dimensions, then E(X + Y ) = E(X) + E(Y ) E(AX) = AE(X) E(AXB) = AE(X)B E(AX + BY ) = AE(X) + BE(Y )
• Further, if c is a scalar-valued constant then E(cX) = cE(X). 74
Mean Vectors and Covariance Matrices • Suppose that X is p × 1 (continuous) random vector drawn from some p−dimensional distribution.
• Each element of X, say Xj has its own marginal distribution with marginal mean µj and variance σjj defined in the usual way: µj = σjj =
Z ∞ −∞ Z −∞
xj fj (xj )dxj (xj − µj )2fj (xj )dxj
75
Mean Vectors and Covariance Matrices (cont’d) • To examine association between a pair of random variables we need to consider their joint distribution.
• A measure of the linear association between pairs of variables is given by the covariance h
σjk = E (Xj − µj )(Xk − µk ) =
Z ∞ Z ∞ −∞ −∞
i
(xj − µj )(xk − µk )fjk (xj , xk )dxj dxk .
76
Mean Vectors and Covariance Matrices (cont’d) • If the joint density function fjk (xj , xk ) can be written as the product of the two marginal densities, e.g., fjk (xj , xk ) = fj (xj )fk (xk ), then Xj and Xk are independent. • More generally, the p−dimensional random vector X has mutually independent elements if the p−dimensional joint density function can be written as the product of the p univariate marginal densities. • If two random variables Xj and Xk are independent, then their covariance is equal to 0. [Converse is not always true.] 77
Mean Vectors and Covariance Matrices (cont’d) • We use µ to denote the p × 1 vector of marginal population means and use Σ to denote the p × p population variance-covariance matrix: i 0 Σ = E (X − µ)(X − µ) . h
• If we carry out the multiplication (outer product)then Σ is equal to:
)2
(X1 − µ1 (X1 − µ1 )(X2 − µ2 ) 2 (X2 − µ2)(X1 − µ1) (X − µ ) 2 2 E ... ... (Xp − µp )(X1 − µ1 ) (Xp − µp )(X2 − µ2 )
··· ··· ··· ···
(X1 − µ1 )(Xp − µp ) (X2 − µ2 )(Xp − µp ) . ... (Xp − µp )2
78
Mean Vectors and Covariance Matrices (cont’d) • By taking expectations element-wise we find that Σ=
σ11 σ12 · · · σ1p σ21 σ22 · · · σ2p ... ... ... ··· σp1 σp2 · · · σpp
.
• Since σjk = σkj for all j 6= k we note that Σ is symmetric.
• Σ is also non-negative definite
79
Correlation Matrix • The population correlation matrix is the p × p matrix with off-diagonal elements equal to ρjk and diagonal elements equal to 1.
1 ρ12 · · · ρ1p ρ21 1 · · · ρ2p . ... ... · · · ... ρp1 ρp2 · · · 1
• Since ρij = ρji the correlation matrix is symmetric • The correlation matrix is also non-negative definite 80
Correlation Matrix (cont’d) • The p × p population standard deviation matrix V 1/2 is a diagonal ma√ trix with σjj along the diagonal and zeros in all off-diagonal positions. Then Σ = V 1/2P V 1/2 and the population correlation matrix is (V 1/2)−1Σ(V 1/2)−1
• Given Σ, we can easily obtain the correlation matrix
81
Partitioning Random vectors • If we partition the random p × 1 vector X into two components X1, X2 of dimensions q × 1 and (p − q) × 1 respectively, then the mean vector and the variance-covariance matrix need to be partitioned accordingly. • Partitioned mean vector: "
E(X) = E
X1 X2
#
"
=
E(X1) E(X2)
#
"
=
µ1
#
µ2
• Partitioned variance-covariance matrix: "
V ar(X1)
#
"
#
Cov(X1, X2) Σ11 Σ12 Σ= = , 0 Cov(X2, X1) V ar(X2) Σ12 Σ22 where Σ11 is q × q, Σ12 is q × (p − q) and Σ22 is (p − q) × (p − q). 82
Partitioning Covariance Matrices (cont’d) • Σ11, Σ22 are the variance-covariance matrices of the sub-vectors X1, X2, respectively. The off-diagonal elements in those two matrices reflect linear associations among elements within each sub-vector.
• There are no variances in Σ12, only covariances. These covariancs reflect linear associations between elements in the two different sub-vectors.
83
Linear Combinations of Random variables • Let X be a p × 1 vector with mean µ and variance covariance matrix Σ, and let c be a p×1 vector of constants. Then the linear combination c0X has mean and variance: E(c0X) = c0µ,
and
V ar(c0X) = c0Σc
• In general, the mean and variance of a q × 1 vector of linear combinations Z = Cq×pXp×1 are µZ = CµX and ΣZ = CΣX C 0.
84
Cauchy-Schwarz Inequality • We will need some of the results below to derive some maximization results later in the course. Cauchy-Schwarz inequality Let b and d be any two p × 1 vectors. Then, (b0d)2 ≤ (b0b)(d0d) with equality only if b = cd for some scalar constant c . Proof: The equality is obvious for b = 0 or d = 0. For other cases, consider b − cd for any constant c 6= 0 . Then if b − cd 6= 0, we have 0 < (b − cd)0(b − cd) = b0b − 2c(b0d) + c2d0d, since b − cd must have positive length. 85
Cauchy-Schwarz Inequality We can add and subtract (b0d)2/(d0d) to obtain 0 < b0b−2c(b0d)+c2d0d−
(b0d)2
d0d
+
(b0d)2
d0d
= b0b−
!2 0 bd
(b0d)2
0 d) c − +( d d0d d0d
Since c can be anything, we can choose c = b0d/d0d. Then, (b0d)2 0 0 0 and we can divide both sides by x0B x to get an upper bound (x0d)2 0 B −1 d. ≤ d x0B x Differentiating the left side with respect to x shows that maximum is attained at x = cB −1d. 88
Maximization of a Quadratic Form on a Unit Sphere • B is positive definite with eigenvalues λ1 ≥ λ2 ≥ · · · ≥ λp ≥ 0 and associated eigenvectors (normalized) e1, e2, · · · , ep. Then
x0B x max 0 = λ1, x6=0 x x
attained when x = e1
x0B x min 0 = λp, x6=0 x x
attained when x = ep.
• Furthermore, for k = 1, 2, · · · , p − 1,
x0B x = λk+1 is attained when x = ek+1. max 0 x⊥e1,e2,··· ,ek x x See proof at end of chapter 2 in the textbook (pages 80-81). 89
