Automatic regression benchmark system

Nicolas Desprès

Description

Automatic regression benchmark system
Nicolas Desprès
LRDE seminar, June 8, 2005, revision 601
http://www.lrde.epita.fr/Copying this document
Copying this document
cCopyright 2005 LRDE.
Permission is granted to copy, distribute and/or modify this document
under the terms of the GNU Free Documentation License, Version 1.2
or any later version published by the Free Software Foundation; with the
Invariant Sections being just “Copying this document”, no Front Cover
Texts, and no Back Cover Texts.
A copy of the license is provided in the file COPYING.DOC.
Automatic regression benchmark system, Nicolas Desprès LRDE seminar, June 8, 2005 1Introduction
Introduction
A regression benchmark system is about keeping track of program
performance through different versions.
Automatic regression benchmark system, Nicolas Desprès LRDE seminar, June 8, 2005 2Table of Contents
Table of Contents
Generalities.......................................................................................... 4
Regression benchmark system ......................................................... 9
Ranch.................................................................................................... 14
Precision............................................................. 21
Automatic regression benchmark system, Nicolas Desprès LRDE seminar, June 8, 2005 3Generalities
Generalities
Automatic regression benchmark system, Nicolas Desprès LRDE seminar, June 8, 2005 4Generalities What does benchmark ...

Sujets

• Séminariste
• Fender Precision Bass
• Intel 4040
• Score de Malinas
• Mesure pour mesure
• Rail

Informations

• Publié par: Ictae
• Nombre de lectures: 62
• Langue: English

Extrait

Automaticregressionbenchmarksystem EDRL NicolasDesprès seminar,June8,2005,revision http://www.lrde.epita.fr/ 106

Copyingthisdocument Copyingthisdocument Copyright c 2005LRDE. Permissionisgrantedtocopy,distributeand/ormodifythisdocument underthetermsoftheGNUFreeDocumentationLicense,Version1.2 oranylaterversionpublishedbytheFreeSoftwareFoundation;withthe InvariantSectionsbeingjust“Copyingthisdocument”,noFront-Cover Texts,andnoBack-CoverTexts. AcopyofthelicenseisprovidedintheleCOPYING.DOC. Automaticregressionbenchmarksystem,NicolasDesprès-LRDEseminar,June8,20051

Introduction Introduction Aregressionbenchmarksystemisabout performancethroughdifferentversions. keeping Automaticregressionbenchmarksystem,NicolasDesprès-LRDEseminar,June8,2005 kcartfoprogram 2

TableofContents TableofContents Generalities.......................................................................................... Regressionbenchmarksystem......................................................... Ranch.................................................................................................... Precision.............................................................................................. Automaticregressionbenchmarksystem,NicolasDesprès-LRDEseminar,June8,2005 4 9 41 12 3

Generalities Automaticregressionbenchmark Generalities system,NicolasDesprès-LRDEseminar,June8,20054

Generalities Abenchmarkis: .1 .2 Whatdoesbenchmarkmeans? Whatdoesbenchmarkmeans? Anypermanentmarktowhichotherlevelsmaybereferred.(dic) Atestorseriesoftestsdesignedtocomparethequalitiesor performanceofdifferentdevicesofthesametype.Certainsetsof computerprogramsaremuchusedasbenchmarksforcomparing theperformanceofdifferentcomputers,especiallybycomparing thetimeittakestocompleteatest.(dic) Automaticregressionbenchmarksystem,NicolasDesprès-LRDEseminar,June8,20055

GeneralitiesDifferentbenchmarkcategories Differentbenchmarkcategories Wecandistinguishdifferentkindsofbenchmarks: • regression, • scalability. Automaticregressionbenchmarksystem,NicolasDesprès-LRDEseminar,June8,20056

Generalities Differentmeasurecategories Severaltypesofmeasures ⇒ severalunits. Wewanttohandledifferentkindsofmeasures: Differentmeasurecategories • Howlongdoestheprogramstayintheusercontext? • Howmuchmemorydoesituse? • Howmanypacketsdoesitsendthroughthenetwork? Howbigarethey? • Howmanypixelsdoesitaccess? • andsoon... Automaticregressionbenchmarksystem,NicolasDesprès-LRDEseminar,June8,20057

GeneralitiesDifferentinputdatatypes/sizes Differentinputdatatypes/sizes Severalinputdatatypes ⇒ severalunits. Aninputsize N=200 maymean: • anautomatonof 200 statesinVaucanson; • animageof 200 × 200 pixelsinOlena. Automaticregressionbenchmarksystem,NicolasDesprès-LRDEseminar,June8,20058

Regressionbenchmarksystem Regressionbenchmark system Automaticregressionbenchmarksystem,NicolasDesprès-LRDEseminar,June8,20059

Regressionbenchmarksystem Wewantto: Motivations • detectperformanceregressionASAP; • automatebenchmarkexecutions; • provideagenericresultanalyzer/visualizer; • centralizeresultstorage. Automaticregressionbenchmarksystem,NicolasDesprès-LRDEseminar,June8,2005 Motivations 01