Distributed learning in sensor networks [Elektronische Ressource] : an online trained spiral recurrent neural network, guided by an evolution framework, making duty cycle reduction more robust / vorgelegt von Huaien Gao

Huaien Gao

Sujets

• Informatik

Informations

• Publié par: ludwig-maximilians-universitat_munchen
• Publié le: 01 janvier 2008
• Nombre de lectures: 13
• Langue: English
• Poids de l'ouvrage: 2 Mo

Extrait

Distributed Learning in Sensor Networks
— an online-trained spiral recurrent neural network, guided by an
evolution framework, making duty-cycle reduction more robust
Huaien Gao
Mu¨nchen 2008Distributed Learning in Sensor Networks
— an online-trained spiral recurrent neural network, guided by an
evolution framework, making duty-cycle reduction more robust
Huaien Gao
Dissertation
an dem Institut fu¨r Informatik
der Ludwig–Maximilians–Universit¨at
Mu¨nchen
vorgelegt von
Huaien Gao
geb. 25.10.1977
Mu¨nchen, den 27.05.2008Erstgutachter: Hans-Peter Kriegel
Zweitgutachter: Darius Burschka
Tag der mu¨ndlichen Pru¨fung: 26.Jan.2009Acknowledgement
ThisthesisiscompletedwithinthejointPh.D.programbetweenthedepartmentofDatabase
and Information Systems in the Institute for Computer Science in the University of Mu-
nich andthe department of Learning System inCorporateTechnology, Siemens AG. Many
people have given valuable advises during the research and the writing. Without their
support, this thesis cannot have been written.
Many thanks are due to Professor Hans-Peter Kriegel who has been supportive not only
on my research activities but also on my application of “Aufenthaltsbewilligung” which is
very important to me. Also thanks are due to Prof. Dr. Darius Burschka for his kindly
agreement to act as the second examiner of this thesis.
I would like to express my sincere gratitude to Dr. Rudolf Sollacher, in Corporate Tech-
nology in Siemens AG, for he has guided, instructed, encourage, inspired and continually
motivated me.
I am also very grateful to Dr. Paul-Theo Pilgram, in Corporate Technology in Siemens
AG, for his generous hospitality to proofread the whole manuscript, and making the thesis
much smoother.
I am indebted to Prof. Dr. Bernd Schu¨rmann and Dr. Thomas Runkler, heads of de-
partment of Learning System in Corporate Technology of Siemens AG, who have been
constantly supportive to my research and given me advises on the thesis.
Prof. Dr. Martin Greiner and Dr. Jochen Cleve are always helpful to me, no matter it
concerns about my research or other difficulty.
Iwill remember thehelpfromcolleagues bothinSiemens AG andinUniversity ofMunich.
The following list is undoubtedly incomplete: Dr. Kai Yu, Anton Maximilian Schaffer,
Dr. Kai Heesche, Dr. Christoph Tietz, Dr. Hans-Georg Zimmermann, Dr. Peter Mayer,
Dr. Peter Kunath, Mrs. Susanne Grienberger, Dr. Volker Tresp, Yi Huang, Dr. Marco
Pellegrino, Mrs. Christa Singer.
More than to anyone else, I own to the constantly love and support from my family. In
every stage of my life, they always encourage, support and understand me. No matter thevi ACKNOWLEDGEMENT
difficulties I am facing, they are the ones who tell me never give up and give me the power
to confront. This thesis is dedicated to them.Abstract
The sensor networks of today tend to be built from “intelligent” sensor nodes. Such nodes
have substantial processing capability and memory, are locally distributed and communi-
cate wirelessly with one another. They are mostly battery-powered, possibly even with a
lifetime battery. Power management is hence a prime concern.
Such intelligent sensors, or “smart sensors”, are typically involved in some diagnosis task,
which wouldbenefit greatlyfromanabilitytopredict environment data. Thebest ofthose
predictions are obtained by training a learning-model with environment data. This task
is non-trivial, computationally intensive and thus expensive on energy, particularly if the
model imposed by the environment data is dynamic and complex. As the training data
usually come from diverse sources, not only from the nearest sensor, the learning node
must communicate with other nodes to get at their measurement data. Data processors
can be made very energy efficient, whereas radio is inherently wasteful. The ultimate aim
is to find the right balance between prediction quality and energy consumption.
Unlike conventional energymanagement solutions, which provide routingmethods orcom-
munication protocols, this thesis introduces an efficient learning algorithm which improves
prediction performance of sensors and reduces computational complexity. It introduces
two techniques which both reduce the overall energy consumption of the sensor network:
intra-node and inter-node solutions.
Intra-node solution: A sensor’s duty cycle is the time fraction during which the sensor
is active. Battery life can be extended by reducing the duty cycle of the sensor. De-
pending on the communication protocol, radio communication coincides with more
or less of the sensor’s active time. The radio duty cycle can be reduced by com-
municating less frequently. This thesis introduces Spiral Recurrent Neural Networks
(SpiralRNN), a novel model for on-line prediction, where some received data get
substituted by predicted data. It is shown that the SpiralRNN model works reliably,
thus opening a way to significant energy savings.
Inter-node solution: Communication between sensor nodes can also be diminished at
network level. Transmitting data (and receiving them) consumes energy and blocks
the airwaves, but the information transmitted is often irrelevant (depending on the
application). This thesis introduces a heuristic evolutionary method, the evolutionviii ABSTRACT
framework, which weighs up energy consumption against prediction performance,
adapting its model structure to the environment data as well as to application con-
straints. The complexity of the model gets lowered by removing some hidden nodes.
The communication effort gets reduced by removing dependencies on various “unim-
portant” data (which makes communication dispensable in those cases).
Spiral Recurrent Neural Networks (SpiralRNN), in combination with duty-cycle reduction
andtheevolution framework,areapowerfultechniqueforbalancingpredictionperformance
against energy consumption, and are hence valuable in the construction of sensor network
applications.Contents
Acknowledgement v
Abstract vii
Contents ix
1 Introduction 1
1.1 Goal . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Challenges and Solutions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.2.1 Intelligence of Sensor Node . . . . . . . . . . . . . . . . . . . . . . . 3
1.2.2 Evolution Framework . . . . . . . . . . . . . . . . . . . . . . . . . . 6
1.3 Structure of Thesis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2 State of the Art 9
2.1 Recurrent Neural Networks. . . . . . . . . . . . . . . . . . . . . . . . . . . 9
2.2 On-line Learning Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . 17
2.2.1 Gradient Descent Learning . . . . . . . . . . . . . . . . . . . . . . . 18
2.2.2 Back-Propagation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
2.2.3 Kalman Filters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
2.2.4 Learning Paradigms . . . . . . . . . . . . . . . . . . . . . . . . . . 21
2.3 Backgrounds on Sensor Network Application . . . . . . . . . . . . . . . . . 22x CONTENTS
2.4 Evolution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
2.4.1 Genetic Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
2.4.2 Evolution Strategies . . . . . . . . . . . . . . . . . . . . . . . . . . 25
2.4.3 Evolution with Neural Networks . . . . . . . . . . . . . . . . . . . . 25
3 Spiral Recurrent Neural Networks 27
3.1 Structure and Eigenvalues . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
3.1.1 Hidden Units . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
3.1.2 SpiralRNNs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
3.1.3 Eigenvalues in SpiralRNNs . . . . . . . . . . . . . . . . . . . . . . . 31
3.2 Implementation of SpiralRNNs . . . . . . . . . . . . . . . . . . . . . . . . 32
3.2.1 The Forward Phase . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
3.2.2 The Training Phase . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
3.2.3 The Autonomous Test Phase. . . . . . . . . . . . . . . . . . . . . . 41
4 Applications with SpiralRNNs 43
4.1 Experimental Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
4.1.1 Tasks. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
4.1.2 Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
4.1.3 Training . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
4.1.4 Testing and Measurement . . . . . . . . . . . . . . . . . . . . . . . 48
4.2 Simulations with Time Series Prediction . . . . . . . . . . . . . . . . . . . 49
4.2.1 Spike21 Dataset. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
4.2.2 Mackey-Glass Dataset . . . . . . . . . . . . . . . . . . . . . . . . . 51
4.2.3 Lorenz Dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
4.2.4 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
4.3 MouseTracking with SpiralRNNs . . . . . . . . . . . . . . . . . . . . . . . 56