Canonical Correlationa Tutorial

Description

Canonical Correlationa TutorialMagnus BorgaJanuary 12, 2001Contents1 About this tutorial 12 Introduction 23 Definition 24 Calculating canonical correlations 35 Relating topics 35.1 The difference between CCA and ordinary correlation analysis . . 35.2 Relationtomutualinformation................... 45.3 Relation to other linear subspace methods ....... 45.4 RelationtoSNR........................... 55.4.1 Equalnoiseenergies... 55.4.2 Correlation between a signal and the corrupted signal . . . 6A Explanations 6A.1Anoteoncorelationandcovariancematrices........... 6A.2Affinetransformations ................. 6A.3Apieceofinformationtheory......... 7A.4 Principal component analysis . ............. 9A.5 Partial least squares .............. 9A.6 Multivariate linear regression . ............. 9A.7Signaltonoiseratio.............. 101 About this tutorialThis is a printable version of a tutorial in HTML format. The tutorial may bemodified at any time as will this version. The latest version of this tutorial isavailable athttp://people.imt.liu.se/˜magnus/cca/.1[wxi][2Tyy[[Ew][2yxxy[:Ew>^ETxyx=]yETwxxiwT^xjTTxC=wxEyyyyyT]^Ewwyyy]T=wwCTxx[Cixyjw=y[qTwxyTwxTCCxxww6xTw=2 IntroductionCanonical correlation ...

Informations

• Publié par: Endri
• Nombre de lectures: 103
• Langue: English

Extrait

Canonical Correlation
a Tutorial
Magnus Borga
January 12, 2001
Contents
1 About this tutorial 1
2 Introduction 2
3 Definition 2
4 Calculating canonical correlations 3
5 Relating topics 3
5.1 The difference between CCA and ordinary correlation analysis . . 3
5.2 Relationtomutualinformation................... 4
5.3 Relation to other linear subspace methods ....... 4
5.4 RelationtoSNR........................... 5
5.4.1 Equalnoiseenergies... 5
5.4.2 Correlation between a signal and the corrupted signal . . . 6
A Explanations 6
A.1Anoteoncorelationandcovariancematrices........... 6
A.2Affinetransformations ................. 6
A.3Apieceofinformationtheory......... 7
A.4 Principal component analysis . ............. 9
A.5 Partial least squares .............. 9
A.6 Multivariate linear regression . ............. 9
A.7Signaltonoiseratio.............. 10
1 About this tutorial
This is a printable version of a tutorial in HTML format. The tutorial may be
modified at any time as will this version. The latest version of this tutorial is
available athttp://people.imt.liu.se/˜magnus/cca/.
12 Introduction
Canonical correlation analysis (CCA) is a way of measuring the linear relationship
between two multidimensional variables. It finds two bases, one for each variable,
that are optimal with respect to correlations and, at the same time, it finds the
corresponding correlations. In other words, it finds the two bases in which the
correlation matrix between the variables is diagonal and the correlations on the
diagonal are maximized. The dimensionality of these new bases is equal to or less
than the smallest dimensionality of the two variables.
An important property of canonical correlations is that they are invariant with
respect to affine transformations of the variables. This is the most important differ-
ence between CCA and ordinary correlation analysis which highly depend on the
basis in which the variables are described.
CCA was developed by H. Hotelling [10]. Although being a standard tool
in statistical analysis, where canonical correlation has been used for example in
economics, medical studies, meteorology and even in classification of malt whisky,
it is surprisingly unknown in the fields of learning and signal processing. Some
exceptionsare[2,13,5,4,14],
For further details and applications in signal processing, see my PhD thesis [3]
and other publications.
3 Definition
Canonical correlation analysis can be defined as the problem of finding two sets of
basis vectors, one for and the other for , such that the correlations between the
projections of the variables onto these basis vectors are mutually maximized.
Let us look at the case where only one pair of basis vectors are sought, namely
the ones corresponding to the largest canonical correlation: Consider the linear
combinations and of the two variables respectively. This
means that the function to be maximized is
(1)
yy
The maximum of with respect to and is the maximum canonical
correlation. The subsequent canonical correlations are uncorrelated for different
solutions, i.e.
]=
=0
]=
=0 for (2)
yj
yy
yj
yi
yi
]=
=0
yj
yj
2
w
w
[
x
w
]
=
E
j
[
[
^
x
w
=
T
w
y
xy
yy
<
T
j
^
xx
w
C
y
i
]
T
=
T
w
[
T
[
x
T
C
8
xy
:
w
i
y
=
q
T
w
w
T
T
x
w
C
[
xx
j
w
[
x
yy
w
x
T
C
y
y
C
i
T
=
w
T
y
w
^
:
C
w
T
^
>
T
>
xy
E
x
x
T
x
w
]
^
E

w
[
xi
E
T
=
xj
]
w
w
xi
x
xx
2
xj
w
E
y
y
y
y
[
]
E
E
]
w
2
T
x
T
[
w
E
w
p
^
]
[
xy
E
[
E
E
x
=
y

]
y
E
w
w
^
xi
T
T
xx
q
x
w
i
xi
=
xy
w
^
y
x
=
6
y
j:
x
y
]The projections onto and ,i.e. and , are called canonical variates.
4 Calculating canonical correlations
Consider two random variables and with zero mean. The total covariance
matrix

(3)
yx
yy
is a block matrix where and are the within sets covariance matrices of
and respectively and is the between sets covariance matrix.
yx
The canonical correlations between and can be found by solving the eigen
value equations
yx
yy
(4)
yx
yy
where the eigenvalues are the squared canonical correlations and the eigen
vectors and are the normalized correlation basis vectors.The
number of non zero solutions to these equations are limited to the smallest dimen
sionality of and . E.g. if the dimensionality of and is 8 and 5 respectively,
the maximum number of canonical correlations is 5.
Only one of the eigenvalue equations needs to be solved since the solutions are
related by
(5)
yx
yy
where
yy
(6)
5 Relating topics
5.1 The difference between CCA and ordinary correlation analysis
Ordinary correlation analysis is dependent on the coordinate system in which the
variables are described. This means that even if there is a very strong linear rela
tionship between two multidimensional signals, this relationship may not be visible
in a ordinary correlation analysis if one coordinate system is used, while in another
coordinate system this linear relationship would give a very high correlation.
CCA finds the coordinate system that is optimal for correlation analysis, and
the eigenvectors of equation 4 defines this coordinate system.
3
C
x
(
w
x
^
w
xx
C
C
xx
x
xy
T
C
w
^
^
"
y
#
w
C
^
C
C
C
y
2
T

w
y
^
xy
s
=
=
x
y

1
xx
:
y
xy
=
x
x
1

w
;
w
y
C
w
=
^
w
C
C
y
1

^
=
y
x

w
C
^

C
E
x

w
y
^
y
xx
T
C
C
x
C

x
=
C
y
=
w
T
^
y
xy
C
C
xx
:
y
<
^
8
x
y
^
x
1
w
^
x
x
w

y
^
x
x
y
1
x
2
y
C
y
xx
C
xy
x
w
=
=

w
C
2
xx
C
Example: Consider two normally distributed two dimensional variables and
with unit variance. Let . It is easy to confirm that the correlation
matrix between and is
(7)
This indicates a relatively weak correlation of 0.5 despite the fact that there is a
perfect linear relationship (in one dimension) between and .
A CCA on this data shows that the largest (and only) canonical correlation is
one and it also gives the direction in which this perfect linear relationship
lies. If the variables are described in the bases given by the canonical correlation
basis vectors (i.e. the eigenvectors of equation 4), the correlation matrix between
the variables is
(8)
5.2 Relation to mutual information
There is a relation between correlation and mutual information. Since informa-
tion is additive for statistically independent variables and the canonical variates
are uncorrelated, the mutual information between and is the sum of mutual
information between the variates and if there are no higher order statistic de
pendencies than correlation (second order statistics). For Gaussian variables this
means
)= (9)
Kay [13] has shown that this relation plus a constant holds for all elliptically sym-
metrical distributions of the form
(10)
5.3 Relation to other linear subspace methods
Instead of the two eigenvalue equations in 4 we can formulate the problem in one
single eigenvalue equation:
A^ (11)
where
and (12)
yx
yy
Solving the eigenproblem in equation 11 with slightly different matrices will
give solutions to principal component analysis (PCA), partial least squares (PLS)
and multivariate linear regression (MLR). The matrices are listed in table 1.
4
xy

2
z
2
(
0
1
:
C
y
T
y
)
5
z
:

Q
z
x
z
y
))
+
:


:
)
y
i
0
2
y

(1
(1

1
y

I
log
x
B
+
1
x
i
2
w
xy
=
:

5
^
0
w
:
X
T
2

1
1
=
w


A

=
i

1
0
log
C
1
xy
;
C
(
)
i
0
i
:
y
x

y
;
1
B
y
=
=

1
C<