Automatic regression benchmark system

Nicolas Desprès

Description

Automaticregressionbenchmarksystem
NicolasDesprès
oTechnicalReportn 0513,June2005
revision 921
Regression benchmark is a part of regression testing that aims at an automatic detection of performance
regrduringapplicationdevelopment. Thegoalistodetectassoonaspossiblethesmallestchangeof
performance. Weneedprecisemeasurementofsmallpartsoftheprogramtoachievethat. Although,many
benchmarksystemsalreadyexistsnonearefullyautomatedand/oradaptedtowiderangeofapplications.
However, automation is a crucial requirement in order to detect regression as soon as possible. This paper
tackles generalities about performance measurements, then gives the requirements of such a system, and
finallyproposesamodeling.
L’évaluation des régressions de performance d’une application fait partie intégrante de la phase de test de
régression.Lebutestd’exprimerleplusprécisémentpossiblelesdifférencesdeperformanceentredeuxver
sions.Parprécision,nousentendonsàlafoislapertinencedel’estimateurdetempsutiliséetlagranularité
despartiesévaluées.
Bienquedenombreuxsystèmesd’évaluationcomparativedesperformancesexistentdéjà,peusontcom
plètementautomatiséset/ouadaptésàdifférentessortesd’applications.Pourtant,l’automatisationdecette
phaseestcrucialeafindedétecterleplustôtpossiblelespertesdeperformance.
Cet exposé présente tout d’abord les prérequis à la mise en place d’un tel système. Puis son architec
ture ainsi que son application au projet Transformers. Ensuite, nous comparerons différentes techniques
d’estimations ...

Sujets

• Linternaute.com
• Régression linéaire
• Arms
• African Institute for Mathematical Sciences
• Printze
• Dépôt (informatique)

Informations

• Publié par: Voin
• Nombre de lectures: 86
• Langue: English

Extrait

Automaticregressionbenchmarksystem NicolasDesprès TechnicalReportn o 0513,June2005 revision921 Regressionbenchmarkisapartofregressiontestingthataimsatanautomaticdetectionofperformance regressionduringapplicationdevelopment.Thegoalistodetectassoonaspossiblethesmallestchangeof performance.Weneedprecisemeasurementofsmallpartsoftheprogramtoachievethat.Although,many benchmarksystemsalreadyexistsnonearefullyautomatedand/oradaptedtowiderangeofapplications. However,automationisacrucialrequirementinordertodetectregressionassoonaspossible.Thispaper tacklesgeneralitiesaboutperformancemeasurements,thengivestherequirementsofsuchasystem,and finallyproposesamodeling. L’évaluationdesrégressionsdeperformanced’uneapplicationfaitpartieintégrantedelaphasedetestde régression.Lebutestd’exprimerleplusprécisémentpossiblelesdifférencesdeperformanceentredeuxver- sions.Parprécision,nousentendonsàlafoislapertinencedel’estimateurdetempsutiliséetlagranularité despartiesévaluées. Bienquedenombreuxsystèmesd’évaluationcomparativedesperformancesexistentdéjà,peusontcom- plètementautomatiséset/ouadaptésàdifférentessortesd’applications.Pourtant,l’automatisationdecette phaseestcrucialeafindedétecterleplustôtpossiblelespertesdeperformance. Cetexposéprésentetoutd’abordlesprérequisàlamiseenplaced’untelsystème.Puissonarchitec- tureainsiquesonapplicationauprojetTransformers.Ensuite,nouscompareronsdifférentestechniques d’estimationsdutempsd’exécution.Etenfin,nousévoqueronslespossibilitésd’adaptationd’untelenvi- ronnementàd’autresorted’applications. Keywords automatic,regressionbenchmark,performanceanalysis,visualization,database,datacollection LaboratoiredeRechercheetDéveloppementdel’Epita 14-16,rueVoltaire–F-94276LeKremlin-Bicêtrecedex–France Tél.[phone]–Fax.[phone] [e-mail] –http://www.lrde.epita.fr/

Copyingthisdocument 2 Copyright c 2005LRDE. Permissionisgrantedtocopy,distributeand/ormodifythisdocumentunderthetermsof theGNUFreeDocumentationLicense,Version1.2oranylaterversionpublishedbytheFree SoftwareFoundation;withtheInvariantSectionsbeingjust“Copyingthisdocument”,noFront- CoverTexts,andnoBack-CoverTexts. AcopyofthelicenseisprovidedinthefileCOPYING.DOC.

Contents 1Introduction 2Requirements 2.1Overview........................................... 2.2Automaticdataacquisition................................ 2.3Unifiedresultsformat................................... 2.4Resultsrepository...................................... 2.5Resultsanalyzes/visualizationinterface......................... 2.6Platform-dependency.................................... 3Design 3.1Overviewoftheregressionbenchmarksystem..................... 3.2Thebenchmarksuite.................................... 3.3Thepopulatescript..................................... 3.4Thecollectscript...................................... 3.5Thewebinterface...................................... 3.6Thedatabase......................................... 3.6.1Tablesdescription................................. 3.6.2Tablealterationversusnumerousrecords.................... 3.7Typicalusecasescenario.................................. 3.8Theunifiedresultsformat................................. 3.8.1Featuredmaterials................................. 3.8.2Chosenlanguage.................................. 4Conclusion 5Bibliography 4 5 5 5 5 6 6 6 7 7 7 8 8 9 9 9 01 01 11 11 21 41 51

Chapter1 Introduction Thistechnicalreportpresentsaregressionbenchmarksystem.Suchasystemaimsatpreventing performanceregressionwhichmaybeintroducedduringaprojectdevelopment. Betweentwomajorversionsofaprogram,manysmalllossesofperformancemaybeintro- ducedcontinuouslybymaintainerswhiletheyaredevelopingtheprogram.Theyoftendonot detectthesesmallperformanceregressionsbecausetheydonotruntheirbenchmarksuiteforev- erypatchtheyapply 1 orbecausetheirperformancemeasurementsarenotaccurateenough.The sumofallthesesmalllossesofperformancemayresultinasignificantperformanceregression. Whenmaintainersdetectanimportantregressionoftheefficiencyoftheirprogram,hundreds ofpatchesarealreadycommitted.Thus,theyareunabletofindwhenthisregressionhappened, especiallyifitisthesumofseveralsmallregressions.Thisproblemcanbeavoidedifthebench- marksuiteisruncontinuouslywhiletheprogramismodified.Itisfrequentthatadeveloper teamdotensofpatchesperday(sometimesmore)ontheirproject.Thus,itisverycumbersome torunthebenchmarksuitemanuallyaftereverypatch,speciallyifittakesalongtimetorunit 2 . Moreovertheamountofbenchmarkresultmayincreasequicklysincethenumberofrevisions ofanaverageprojectisoftenaround500.Thus,aregressionbenchmarksystemmustbefully automaticinordertonotoverloadthewholedevelopmentcycle. Thisreportdescribestheregressionbenchmarksystemunderdevelopmentinourlaboratory. Thissystemaimsatsolvingtheproblemintroducedabove.Itisstilladraftbutthemainmodule specification,intermsofrequirementsanddesign,aredefined. Firstofall,therequirementsofthesystemaredetailed.Then,thewholeproject’sarchitecture isdescribed. 1 Numerousprojectsdonotevenhaveanybenchmarksuite. 2 Thisoftenhappenssincethetestsuitemustberunbeforethebenchmarksuiteandthatbothsuitemaybelargerand largerastheprogramgrows.

Chapter2 Requirements Thischaptercoversthespecificationoftherequirementsofanautomaticregressionbenchmark system.Firstofall,wegiveashortdescriptionofallofthemasanoverview.Then,wedetail eachofthemindividually. 2.1Overview Theregressionbenchmarkframeworkmustfulfillthefollowingrequirements[Coursonetal. (2000);Kalibera(2004);Kaliberaetal.(2004)]: • Itmustperformandcollectthemeasurementresultsautomatically. • Itmustprovideaunifiedresultformat. •Itmustmanagearesultrepository. •Itmustfeatureauser-friendlyinterfaceforresultanalysisand/orvisualization. • Itmustbeplatform-independent. Alltheserequirementsaredetailedinthefollowingsections. 2.2Automaticdataacquisition Asmentionedintheintroduction,weaimatmeasuringperformancesforeveryrevisionofa project,inordertodetectperformanceregressionassoonaspossible.Becauseperformancemea- surementsmaytakealongtime,andbecauseitiscommontocommitchangesmorethanten timesperday,theperformancedataacquisitionforagivenprojectisverytimeconsumingand thuscan’tbeperformedmanually.Itiscrucialthattheentirebenchmarkprocessisperformedau- tomatically:fromtheprogramandbenchmarkenvironmentinstallationandrun,totheaddition oftheresultsintotherepository. 2.3Unifiedresultsformat Abenchmarkconsistsinmakingacomparisonoftwomeasurements.Thecomparisonmaybe doneagainstanothercontestantprogramoranolderversionofthesameprogram.Inordertodo suchacomparison,theresultmustbestoredusingthesameformattoavoidtheuseofconverter. Moreover,wewanttobeabletobenchmarksubpartsofabenchmark.So,weneedaresult formatthatsupportsnestedstructures.However,itisoutofthescopeofourprojecttowritea complexparserandacomplexprettyprinter.Thus,weneedaformatwhichiseasytoreadand writefromtheperspectiveofascript.

2.4Resultsrepository6 2.4Resultsrepository Theamountofcollecteddatamayincreasequicklybecause,wewanttoperformmeasurement foreveryrevision.So,weneedastrongstoragesystem(e.g.notaregularfile).Moreover,we needtobeabletosearchandgroupbenchmarkstogetherwhenweanalyzethedata.These analyzesmustbefastenoughandsupportscalability. 2.5Resultsanalyzes/visualizationinterface Theresultanalyze/visualizationinterfacemustprovideaneasyanduser-friendlywaytogen- erategraphsbasedonthecollecteddata.Themostimportantgraphweneedinaregression benchmarksystemistheonerepresentingtheperformanceevolutioninrespecttotherevision numberoftheprogram.Wealsoneedtocomparedifferentprograms:typicallyourprogramand itscontestants. 2.6Platform-dependency Theend-usermaywishtoseethedifferenceofperformanceofitsprogramfromanarchitecture toanother.So,thebenchmarkenvironmentmustbeabletorunondifferentarchitectures.This constraintisappliedespeciallyonthebenchmarksuitewrittenbytheprojectauthors.Mostof, theprojectundertestiscompatiblewiththearchitecturethebenchmarksuiteiscompatibletoo. Thetaskoftheregressionbenchmarksystemisonlytorunthebenchmarksuiteandtocollect theresult.

Chapter3 Design Inthischapter,wepresentthedesignchosentodeveloptheregressionbenchmarksystem.First ofall,wegiveanoverviewofthewholesystem.Secondly,thestructureofthedatabaseisde- tailed.Then,wedescribethetypicalusecasescenario.Finally,wedetailoursuggestionfora unifiedresultformat.Finally,wearguequicklythetoolsetwehavechosentoimplementit. 3.1Overviewoftheregressionbenchmarksystem Thesystemiscomposedofseveralcomponentslistedbelow: • Abenchmarksuite. • Apopulatescript. •Acollectscript. • Awebinterface. •Adatabase. Therelationshipsbetweeneachcomponentareshownonthefigure3.1 1 (onpage8). 3.2Thebenchmarksuite Thebenchmarksuiteisthepartofthesystemwhichactuallyperformsthemeasurements.On thefigure3.1(page8),wecallthispartoftheframework:thebencher. Mostoftheexistingprojectsimplementtheirbenchmarksuitebywritingatestsuitededicated toemphasistheperformancesoftheprograminsteadofthecorrectbehaviorofitsfeatures. Developersaregenerallyinterestedinmeasuringtheamountoftimeand/orthememoryus- agetheirprogramneeds.Thesetwovaluesareeasilycomputablebymeansofareusableextern program(suchastimeorvalgrind[Nethercote(2004)]).Thesetoolsareconvenientbecause theyarenotintrusiveintheprogramcode.Inotherword,theyratherneedmeaningfultestsuites thancodeinstrumentationsinordertoberelevant. Manyprojectsalsoneedmorespecificinformation.Forinstance,ourprojectOlena[Duret-Lutz (2000)],animageprocessinglibrary,cancomputethenumberoftimesanalgorithmaccesses toapixelofanimage.Thisinformationisveryinterestinginordertooptimizeanalgorithm. Contrarytothetimeandmemoryusagevalues,thecomputationofsuchavalueimpliestoin- strumentthelibrarycode.Thisexampleillustratesthatitisveryhardwithcommonlanguages 1 TheCRUDabbreviationistheCreate,Read,UpdateandDeleteactionssequencethatisusuallyperformedona database.

3.3Thepopulatescript Figure3.1:Regressionbenchmarksystemoverview 8 todevelopagenerictoolsthatcanhelptocomputeanymeasureonemayneed.That’stherea- sonwhyourregressionbenchmarksystemdoesnotfeatureagenerictooltoeasethewritingof benchmarkmeasurements.Nevertheless,thispointistackledin[Kaliberaetal.(2004)]. However,asdiscussedinthepreviouschapter,wewanttounifytheformatusedbythebench- marksuitetoprintitsresults.Thisformatisnotonlyamatteroflayout.Itassertsthatnecessary informationispresent.Theinformationisthedescriptionofeverybenchmarkoftheprojectand theirresults.Thedescriptionmustbenonambiguous,inordertoensurethattheinsertionof theresultinthedatabasewon’tneedanyhumaninteractionsduringtheentireprocess.Thema- terialsprovidedtohelpthedeveloperstoprinttherightinformationusingtherightformatis describedinthesectiondedicatedtotheunifiedresultsformat3.8onpage11.Thismaterialis providedasalibraryanditisgeneratedfromtheinformationcontainedinthedatabase. 3.3Thepopulatescript Thegoalofthepopulatescriptistotakethebenchmarkresults(writtenusingtheunifiedresults format)asinputandpopulatethemintothedatabase.Thisscriptiscalledeitherbythecollect script(seesection3.4)orbyahumanoperator. Thebenchmarksuiteofagivenprojectmayberunwithoutourprojectinstalledonthema- chine.That’swhythemodulethatcommittheinformationintoourdatabaseisembeddedinto thepopulatescriptinsteadofthebenchmarksuite. Moreover,incaseofanunexpectedambiguousbenchmarkdescriptionwhichcouldnotbe committedforsanityreasonsintothedatabase,thepopulatescriptkeepstrackoftheresultuntil ahumanintervention.Thus,theautomaticprocessisnotinterruptedandnodataarelost. 3.4Thecollectscript Thegoalofthecollectscriptistoperiodicallyrunthebenchmarksuiteofeveryrevisionofev- eryprojectregisteredinthedatabaseandoneveryconfigurationmentioned.Thus,itfillsthe databaseandensuresthatnonerevisionmeasuresaremissing.

9Design 3.5Thewebinterface Thewebinterfaceallowstheusertodrawgraphsandchartsbasedonthemeasuresstoredinthe database.WehavechosenawebbasedapplicationinsteadofanXwindowoneforportability reasons. 3.6Thedatabase Thedatabaseisthecornerstoneofthesystem.Itisdesignedtoavoidduplicatedfieldandto supporteverybenchmarktype.Thefigure3.2(page9)representstherelationshipbetweenthe tablesofthedatabase. Figure3.2:Theresultsdatabasedescription 3.6.1Tablesdescription Thecentraltable,calledbenchs,storeseverysinglebenchmark.Inthisdatabase,abenchmark representsonemeasureofonefeatureofoneproject.Ameasureisqualifiedbyatypeandascale. Themeasurementtypetellsus,forinstance,ifthebenchmarkmeasuresthememoryusageor thedurationoftheprogram.Thus,thetypehasaunitsuchasbytesorseconds. Themeasurementscalecodestheprograminputsizeusedbythebenchmark.Thisallowsus toperformscalabilitybenchmark.Ascaleisalsoqualifiedbyaunit. Thebenchstablehasthefollowingfields:anid,aname,aprojectid,atypeid,ascaleid,etc... Becauseabenchmarkmaynotbeavailablefromthebeginning(firstrevision)ofaprojectto itsend,therearetwomorefieldscalledstart_revisionandstop_revision.Theyindicate betweenwhichrevisionintervalthebenchmarkmaybeperform. Theexecutionstablestorestheresultofabenchmarkcollectedforeveryvalidrevisionand everyavailableconfiguration.Thebenchmarkismentionedinthistablebymeansofitsidinthe benchstable.Theexecutionstableallowustoeasilychecktheperformanceregressionsfora givenbenchmarkofaproject.

3.7Typicalusecasescenario01 3.6.2Tablealterationversusnumerousrecords Wehavedesignedthisdatabasetoavoidhavingtoalteratablewhilethesystemisrunning.We havealsopaidattentiontonotduplicatedata.Thus,thetablerelationshipsmayseemcomplex, butitisnotrelevantsinceitismaintainedbythesystemandnotbytheusers.Currently,we prefertohaveatablewithmanyrecordsinsteadofcreatingnewtablesonthefly. 3.7Typicalusecasescenario Thetypicalusecasescenarioisshownonthefigure3.3andisdetailedhere: Figure3.3:Atypicalusecasescenario 1.Registeranewbenchmarkinthedatabaseviathewebinterface.Thisincludestheaddition ofnecessarynewtypesorscalesorunits. 2.Askthesystemtoregeneratethebenchmarkconfigurationfile.Basically,thisfilefeatures materialstohelpthedevelopertoprinttheresultsusingtheexpectedformat. 3.Writethecodeneededofthenewbenchmarkintheproject’sbenchmarksuite. 4.Runagainthebenchmarksuiteandredirecttheresultstothepopulatesuite.Thisstageis optionalsincepeoplemaywaitfortheperiodicalbenchmarkexecution. 5.Finally,oncethepopulationprocessisfinishedyoucanwatchtheresultsbymeansofcharts usingthewebinterface.

11Design 3.8Theunifiedresultsformat Thisformataimsatrepresentingeverybenchmarkresultstype.Itis,so-called,unifiedbecause everyprojectmayuseitastheoutputformatoftheirbenchmarksuite. 3.8.1Featuredmaterials Thedevelopersofthebenchmarksuiteshouldnotknowtheunifiedformatthatweprovide.First ofall,itmaybeverycumbersomeforthemtoprintitproperly.Secondly,ifwechangeit,they willhavetoadapttheirbenchmarksuite.Thirdly,theydon’thavetoknowalltheinformation werequiretodescribewithoutambiguityabenchmarkanditsresult. So,weprovidealibrarywhichcontainsalltheinformationneededwhichareregisteredinthe databaseforagivenproject.Basically,thebenchmarksname,typeandunitareavailable.Then, thelibraryinterfacefeaturesmainlytwofunctionswiththefollowingprototypes: voidbegin_benchmark(constchar*name, constchar*type, constchar*unit); voidend_benchmark(doublescore); Forinstance,thedevelopersofthebenchmarksuitemayusethesefunctionsthisway: #include"benchmark.h" staticdoubledo_the_bench(void) { doublescore; /*computethevalueofthescorevariable...*/ returnthe_score; } intmain(void) { doublescore; begin_project(MY_PROJECT); begin_benchmark(MY_BENCH_FOO,MY_TYPE_BAR,MY_INPUT_BAZ); score=do_the_benchmark(); end_benchmark(score); end_project(); return0; } Figure3.4:Anexampleofabenchmarkcode Thebegin_benchmarkfunctionprintsthedescriptionofthebenchmarkwhichisgoingtobe run.Thedescriptionistheidofthebenchmarkinthebenchstableofthedatabase.Thisidis computedbythehashoftheconcatenationofthebenchmarkname,typeandinputstrings.Since thetypenameandinputnamearekeptuniqueinthedatabaseandthebenchmarknamefora givenprojectaswell,therearenoambiguities.Aftercallingthebegin_benchmarkfunction, wereallycomputeourbenchmarkandthen,givethescoreasargumenttotheend_benchmark functioncall.The begin_project functioncallindicatesthatthebenchmarkswrittenuntil