Multi-class recognition using pair-wise classifiers ; Daugelio klasių atpažinimas naudojant klasifikatorius poroms

Rimantas Jankauskas

Sujets

• Informatics
• Multiclass classification

Informations

• Publié par: vilnius_university
• Publié le: 01 janvier 2010
• Nombre de lectures: 13
• Langue: English
• Poids de l'ouvrage: 1 Mo

Extrait

VILNIUS UNIVERSITY





Rimantas Kybartas



MULTI-CLASS RECOGNITION USING PAIR-WISE CLASSIFIERS





Doctoral dissertation
Physical sciences, informatics (09 P)









Vilnius, 2010


Dissertation work was carried out at Vilnius University from 2005 to 2010.


Scientific supervisor:

prof. habil. dr. Šarūnas Raudys (Vilnius University, physical sciences,
informatics – 09 P)


VILNIAUS UNIVERSITETAS





Rimantas Kybartas



DAUGELIO KLASIŲ ATPAŢINIMAS NAUDOJANT KLASIFIKATORIUS
POROMS





Daktaro disertacija
Fiziniai mokslai, informatika (09 P)









Vilnius, 2010

Disertacija rengta 2005 – 2010 metais Vilniaus universitete.


Mokslinis vadovas:

prof. habil. dr. Šarūnas Raudys (Vilniaus universitetas, fiziniai mokslai,
informatika – 09 P)


Acknowledgements

Firstly I would like to thank people who lead me through the astonishing
path of all my studies. Thanks go to my teacher of mathematics at school, Mrs.
Jūratė Raščiauskienė, for trusting in me when I did not. To dr. Ričardas
Kudţma, my teacher at university, who made me trust in myself. To dr.
Antanas Mitašiūnas who supported that trust. And especially I would like to
thank my supervisor prof. habil. dr. Šarūnas Raudys who did not trust in me
when I trusted in myself too much.
My heartiest thanks go to my family (especially to my wife Neringa) who
sacrificed a lot for my PhD studies.
I also thank my colleagues Virginija Daukutė and Viktoras Golubevas
from the Bank of Lithuania for their understanding, support and help.
Thanks to all others who supported and helped me during my PhD
journey.
v
Table of Contents
Acknowledgements........................................................................................................ v
Notation ..................................................... viii
Abbreviations ............... ix
List of Figures ................ x
List of Tables ............................................................................... xi
1. Introduction................................................ 1
1.1 Research Area ................................................................. 1
1.2 Problem Relevance ......................... 1
1.3 Research Object .............................................................. 3
1.4 The Objectives and Tasks of the Research ..................................................... 3
1.5 Research Methods ........................... 4
1.6 Scientific Novelty 4
1.7 Practical Significance of the Work ................................................................. 5
1.8 Defended Theses of Dissertation .... 5
1.9 Approval of Research Results ......... 6
1.10 Thesis Structure ........................................................... 6
2. Multi-class Classification .......................................................... 8
2.1 The Multi-class Classification Task ................................ 8
2.2 Multi-class Classifiers ..................... 9
2.2.1 Artificial Neural Network ........................................ 9
2.2.2 The Radial Basis Function Neural Networks ........................................ 11
2.2.3 Kernel Discriminant Analysis 12
2.3 Fusion of Multi-class Classifiers .. 12
2.3.1 Behavior Knowledge Space ................................... 13
2.3.2 Fuzzy Templates .................................................... 13
2.3.3 Pair-wise Fusion Matrix ........................................ 14
2.4 Concluding Remarks ..................... 15
3. Two Stage Pair-wise Based Classification .............................................................. 16
3.1 Superiority of the Pair-wise Classifiers ........................ 16
3.2 Statistical Classifiers and Their Implementation .......................................... 19
3.3 Pair-wise Crs ...................................................... 20
3.3.1 Single Layer Perceptron ........ 21
3.3.2 Support Vector Classifier ...................................... 23
3.3.3 Decision Trees ....................... 24
3.3.4 Noise Injection ................................................................ 25
3.4 Pair-wise Classifier Fusion Methods and Their Implementations ................ 26
3.5 Non-trainable Pair-wise Fusion Rules .......................... 26
3.5.1 Voting .................................................................................................... 26
3.5.2 Directed Acyclic Graph ......... 27
3.5.3 The Quick Weighted Voting .. 27
3.6 Trainable Pair-wise Fusion Rules . 28
3.6.1 Hastie-Tibshirani ................................................................................... 28
3.6.2 Wu, Lin and Weng ................. 30
3.6.3 Resemblance Model............... 32
3.7 Kind of Pair-wise Classifier Output .............................................................. 33
3.8 Consideration for Fusion Methods‟ Choosing .............................................. 34
3.9 Pair-wise Fuzzy Templates Method 35
vi
3.9.1 Reasoning for a New Method ................................................................ 35
3.9.2 Description of the Pair-wise Fuzzy Templates Method ........................ 35
3.9.3 Weaknesses and Strengths of Pair-wise Fuzzy Templates Method....... 36
3.10 Concluding Remarks ................................................................................. 39
4. On the Issues of Multi-class Classification Task ..................... 41
4.1 Small Sample Size Problem .......... 41
4.2 Generalization Error of the Fisher Classifier ................ 44
4.3 Small Sample Size Solution .......................................................................... 46
4.4 The Unbalanced Sample Size ....... 47
4.4.1 Enhancement of Single-stage K-category SLP-based Neural Net for
Classification ....................................... 49
4.5 Concluding Remarks ..................................................................................... 51
5. Experimental Issues and Results of Classifier Comparison .................................... 52
5.1 The Effect of Simplified Performance Measures and Sample Size on Fusion
Accuracy of the Pair-wise Classifiers ...................................... 52
5.2 The Importance of the Number of Experiments ........... 58
5.3 Experimental Comparison of Fusion Rules .................................................. 61
5.3.1 Data ........................................ 61
5.3.2 Data Whitening ...................................................... 63
5.3.3 Procedure of Experiments...................................................................... 64
5.3.4 Results ... 64
5.4 Bias Reduction in Fusion of Pair-wise Decisions ......... 66
5.5 Concluding Remarks ..................................................................................... 69
6. Application to Mineral Classification ...... 71
6.1 Domain Task ................................. 71
6.2 Data ............................................... 72
6.2.1 Similarity Features ................................................................................. 74
6.3 Practical Results ............................ 74
6.3.1 Reliability of Results ............. 77
6.4 Discussions on Practical Application ................................ 78
6.5 Concluding remarks ...................................................... 79
7. Conclusions ............................................................................. 81
7.1 Recommendations for Multi-class Classification Task Designers ............... 81
7.2 Main Conclusions ......................... 82
7.3 Other Results ................................................................................................. 83
References.................................................... 85

vii
Notation
C – regularization parameter fr support vector classifiers
C – classifier classifying classes  and  . i,j i j
d – dimension of the data.
K – the number of classes.
m – mean vector
N – the size of whole data set.
N – the number of experiments. e
Ni – the size of class  . i
N – the amount of vectors in the training data set. Tr
N – the amount of vectors in the testing data set. Ts
thp – a posteriori probability of the i class i
P – classification error.
thq – a priori probability of the i class i
S – estimate of the sample covariance matrix
v – weight vector of classifier
w – bias weight of classifier 0
 – data set of class i i
 - correlation coefficient
 - covariance matrix
viii
Abbreviations

A-B – Anderson - Bahadur (linear discriminant function)
ANN – artificial neural network
BKS – behavior knowledge space method
CM – covariance matrix
DAG – classifier fusion algorithm using direct acyclic graph
DF – discriminant function
EDC – Euclidean distance classifier
GCCM – data model containing data with Gaussian distribution and sharing
the