A performance test design method and its implementation patterns for multi-services systems [Elektronische Ressource] / vorgelegt von George Din

George Din

Sujets

• Informatik
• Handel

Informations

• Publié par: technische_universitat_berlin
• Publié le: 01 janvier 2009
• Nombre de lectures: 28
• Langue: English
• Poids de l'ouvrage: 4 Mo

Extrait

APerformanceTestDesignMethodanditsImplementation
PatternsforMulti-ServicesSystems
vorgelegt von
Diplom-Ingenieur
George Din
VonderFakultätIV-ElektrotechnikundInformatik
derTechnischenUniversitätBerlin
zurErlangungdesakademischenGrades
DoktorderIngenieurwissenschaften
Dr. Ing.
genehmigteDissertation
Promotionsausschuss:
Vorsitzender: Prof. Dr. AnjaFeldmann
Berichter: Prof. Dr. -Ing. InaSchieferdecker
Berichter: Prof. Dr. phil.-nat. JensGrabowski
TagderwissenschaftlichenAussprache: 8.09.2008
Berlin 2009
D 832Abstract
Over the last few years, the scope of telecommunication services has increased dramatically, mak-
ing network infrastructure-related services a very competitive market. Additionally, the traditional
telecoms are now using Internet technology to provide a larger range of services. The obvious
outcome, is the increase in the number of subscribers and services demanded.
Due to this complexity, the performance testing of continuously evolving telecommunication ser-
vices has become a real challenge. More ecient and more powerful testing solutions are needed.
This ability depends highly on the workload design and on the ecient use of hardware resources
for test execution.
The performance testing of telecommunication services raises an interesting problem: how to cre-
ate adequate workloads to test the performance of such systems. Traditional workload characteri-
zation methods, based on requests/second, are not appropriate since they do not use proper models
for trac composition. In these environments, users interact with the network through consecutive
requests, called transactions. Several transactions create a dialog. A user may demand in parallel
two or more services and dierent behavioural patterns can be observed for dierent groups of
users.
This thesis proposes a performance testing methodology which copes with the afore mentioned
characteristics. The methodology consists of a set of methods and patterns to realize adequate
workloads for multi-service systems. The eectiveness of this methodology is demonstrated
throughout a case study on IP Multimedia Subsystem performance testing.
34Zusammenfassung
In den letzten Jahren hat sich das Angebot an Telekommunikationsdiensten erweitert, was dazu
geführt hat, dass der Markt für Dienste, die sich auf Netzwerkinfrastrukturen beziehen, mittler-
weile sehr umkämpft ist. Ausserdem werden die traditionellen Telekommunikationssysteme mit
Internet Technologien kombiniert, um eine grössere Auswahl an Diensten anbieten zu können.
Daraus resultieren oensichtlich eine Zunahme der Teilnehmeranzahl und ein erhöhter Dienst-
Bedarf.
Infolge dieser Komplexität werden Leistungstests der sich kontinuierlich entwickelnden Telekom-
munikationsdienste zu einer echten Herausforderung. Ezientere und leistungsfähigere Testlö-
sungen werden benötigt. Das Leistungsvermögen hängt ab vom Workload Design und von der
ezienten Nutzung der Hardware für die Testdurchführung.
Die Leistungstests der Telekommunikationsdienste führen zu einer interessanten Problemstel-
lung: Wie soll man adäquate Lastprofile erstellen, um die Leistung solcher Systeme zu testen?
Traditionelle Methoden zur Darstellung der Last, die auf Anfrage/Sekunde basieren, sind nicht
zweckmässig, da sie keine geeigneten Modelle zur Anordnung des Datenverkehrs nutzen. In
diesen Umgebungen soll der Nutzer mit dem Netzwerk über fortlaufende Anfragen, sogenannten
Transaktionen, interagieren. Mehrere Transaktionen erzeugen einen Dialog. Ein Benutzer kann
gleichzeitig zwei oder mehrere Dienste abrufen und es können verschieden Navigationsmuster für
verschiedene Benutzergruppen beobachtet werden.
Diese Promotion schlägt eine Methodologie für Leistungstests vor, die sich mit den vorher genan-
nten Charakteristika beschäftigt. Diese Methodologie setzt sich aus Verfahrensweisen und Mod-
ellen zusammen, die eine adäquate Last von Multi-Dienst Systemen realisieren sollen. Die Leis-
tungsfähigkeit dieser Methodologie wird in einer Fallstudie nachgewiesen, die sich mit Leistung-
stests von IMS-Systemen (IP Multimedia Subsystem) befasst.
56Acknowledgements
The work in this thesis required a large eort on my part, but this eort would not have been
possible without the support of many people. My most special thanks are for Bianca for her love
and patience with my endless working days. I warmly thank my parents for their love, education
and accepting my living away.
I especially thank my advisers Professor Dr. Ina Schieferdecker and Professor Dr. Jens Grabowski.
I wrote a thesis related to TTCN-3 and have been coordinated by the creators of this language, I
cannot imagine ever topping that. I thank Inaer for giving me the opportunity to
work in TTmex and IMS Benchmarking projects and for supporting my work and my ideas over
the years. I also thank her for the many discussions we had and for guiding me over the years to
achieve always the best results. Her knowledge, suggestions and numerous reviews contributed
much to the results and the form of this thesis. I thank Jens Graboswki for the valuable suggestions
and for insuating me a high level of quality and professionalism.
I would like to express my gratitude to Professor Dr. Radu Popescu Zeletin for providing me with
excellent working conditions during my stay at Fraunhofer FOKUS and for giving me the advice
to start a career on testing. My sincere thanks are also due to Professor Dr. Valentin Cristea and
Ivonne Nicolescu for advicing me during my studentship to follow an academical direction.
I also thank the members of the TIP research group at FOKUS (later MOTION-Testing group) for
many fruitful discussions and for providing the environment in both technical and non-technical
sense that made this work not just possible but even enjoyable. Sincere thanks to my colleagues
Diana Vega and Razvan Petre for helping me during the implementation of the software. Also
thank Diana Vega and Justyna Zander-Nowicka for the tireless eorts of proof-reading the docu-
ment. I am grateful to Zhen Ru Dai (aka Lulu), Justyna Zander-Nowicka and Axel Rennoch who
didn’t let me forget that live is more than writing a PhD thesis.
I consider myself fortunate to be involved in two technical challenging projects. I greately enjoyed
working with Theofanis Vassiliou Gioles, Stephan Pietch, Dimitrios Apostolidis and Valentin Za-
harescu during the TTmex project. Many thanks I owe to the colleagues in the IMS Benchmark-
ing project: Tony Roug, Neal Oliver, Olivier Jacques, Dragos Vingarzan, Andreas Homann,
Luc Provoost and Patrice Buriez for their eorts, ideas and debates about IMS benchmarking that
sharpened my arguments. Also especial thanks to INTEL for providing me with the latest hard-
ware technology for doing the experimental work. These experiences were the true definition of
getting a doctorate to me.
78Contents
1 Introduction 19
1.1 Scope of the Thesis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
1.2 Structure of the Thesis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
1.3 Dependencies of Chapters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
2 Fundamentals of Performance Testing 25
2.1 Concepts of Performance Testing . . . . . . . . . . . . . . . . . . . . . . . . . . 25
2.1.1 Testing Process . . . . . . . . . . . . . . . . . . . . . . . . 26
2.1.2 Workload Characterisation . . . . . . . . . . . . . . . . . . . . . . . . . 27
2.1.3 Performance Test Procedures for Dierent Performance Test Types . . . 29
2.1.4 Measurements and Performance Metrics . . . . . . . . . . 30
2.1.5 Performance Test Architectures . . . . . . . . . . . . . . . . . . . . . . 30
2.2 Performance Test Framework . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
2.2.1 The Functional Architecture of a Test Framework . . . . . . . . . . . . . 32
2.2.2 Towards Test System Performance . . . . . . . . . . . . . . . . . . . . . 35
2.3 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
3 Performance Testing Methodology and Realisation Patterns 39
3.1 Multi-Service Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
3.2 Performance Testing Reusing Functional Tests . . . . . . . . . . . . . . . . . . . 40
3.3 The Test Design Process . . . . . . . . . . . . . . . . . . . . . . . 40
3.4 The Performance Test Concepts . . . . . . . . . . . . . . . . . . . . . . . . . . 42
3.4.1 Use-Cases and Scenarios . . . . . . . . . . . . . . . . . . . . . . . . . . 42
3.4.2 Design Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
3.4.3 Trac Set Composition . . . . . . . . . . . . . . . . . . . . . . . . . . 46
3.4.4 Trac-Time Profile . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
3.4.5 Scenario based Performance Metrics . . . . . . . . . . . . . . . . . . . . 50
3.4.6 Global Performance Metrics . . . . . . . . . . . . . . . . . . . . . . . . 52
910 CONTENTS
3.4.7 Design Objective Capacity Definition . . . . . . . . . . . . . . . . . . . 52
3.4.8 Performance Test Procedure . . . . . . . . . . . . . . . . . . . . . . . . 53
3.4.9 Test Report . . . . . . . . . . . . . . . . . . . . . . . . . . 54
3.5 Workload Implementation Patterns Catalogue . . . . . . . . . . . . .