WikiBench: A distributed, Wikipedia based web application ...

Description

WikiBench: A distributed, Wikipedia
based web application benchmark
Master thesis by Erik-Jan van Baaren
Student number 1278967
[e-mail]
Under the supervision of:
Guillaume Pierre
Guido Urdaneta
Vrije Univesiteit Amsterdam
Department of Computer Science
May 13, 2009Abstract
Many di erent, novel approaches have been taken to improve throughput
and scalability of distributed web application hosting systems and relational
databases. Yet there are only a limited number of web application bench-
marks available. We present the design and implementation of WikiBench,
a distributed web application benchmarking tool based on Wikipedia. Wik-
iBench is a trace based benchmark, able to create realistic workloads with
thousands of requests per second to any system hosting the freely available
Wikipedia data and software. We obtained completely anonymized, sam-
pled access traces from the Wikimedia Foundation, and we created software
to process these traces in order to reduce the intensity of its tra c while
still maintaining the most important properties such as inter-arrival times
and distribution of page popularity. This makes WikiBench usable for both
small and large scale benchmarks. Initial benchmarks show a regular day of
tra c with its ups and downs. By using median response times, we are able
to show the e ects of increasing tra c intensities on our system under test.Contents
1 Introduction 2
2 Related Work 4
2.1 TPC-W . . . . . . . . . . . . . . . . . . . . . . . ...

Sujets

• Tracé digital
• Horodatage
• Échantillonnage (musique)
• Ratio de Sharpe
• Controller area network
• Réalisme fantastique

Informations

• Publié par: Anji
• Nombre de lectures: 148
• Langue: English

Extrait

W b i a ki s B ed en w c e h: ba A p d p i li s c tr a i t b io ut n ed b , en W c i h k m ip a e r d k ia MasterthesisbyErik-JanvanBaaren Studentnumber1278967 [e-mail] Underthesupervisionof: GuillaumePierre GuidoUrdaneta VrijeUnivesiteitAmsterdam DepartmentofComputerScience May13,2009

Abstract Manydifferent,novelapproacheshavebeentakentoimprovethroughput andscalabilityofdistributedwebapplicationhostingsystemsandrelational databases.Yetthereareonlyalimitednumberofwebapplicationbench- marksavailable.WepresentthedesignandimplementationofWikiBench, adistributedwebapplicationbenchmarkingtoolbasedonWikipedia.Wik- iBenchisatracebasedbenchmark,abletocreaterealisticworkloadswith thousandsofrequestspersecondtoanysystemhostingthefreelyavailable Wikipediadataandsoftware.Weobtainedcompletelyanonymized,sam- pledaccesstracesfromtheWikimediaFoundation,andwecreatedsoftware toprocessthesetracesinordertoreducetheintensityofitstrafficwhile stillmaintainingthemostimportantpropertiessuchasinter-arrivaltimes anddistributionofpagepopularity.ThismakesWikiBenchusableforboth smallandlargescalebenchmarks.Initialbenchmarksshowaregulardayof trafficwithitsupsanddowns.Byusingmedianresponsetimes,weareable toshowtheeffectsofincreasingtrafficintensitiesonoursystemundertest.

Contents 1Introduction2 2RelatedWork4 2.1TPC-W..............................4 2.2WebPolygraph..........................6 3SystemModel8 3.1Requirements...........................9 3.2WikiBenchdesign.........................11 3.3TraceBenchDesign........................15 3.4WikiBenchWorkflow.......................16 4WorkloadCreation19 4.1ChangingtheRequestRate...................19 4.2AHybridApproach........................21 5BenchmarkResults32 6FutureWork27 6.1ScalinguptrafficandFlashCrowds...............27 6.2Adjustread/writeratio......................27 6.3Adjustthedistributionofpagepopularity...........27 6.4Indicationofrealism.......................28 6.5Moreadvancededits.......................28 7Conclusion 1 92

1Introduction Althoughoriginallyaplacedesignedforresearcherstoeasilyexchangein- formation,theworldwidewebhasbecomeoneofthemostimportantinfor- mationinfrastructuresofmodernsociety.Wehaveseenarapidtransition fromstaticHTMLdocumentstoadvancedwebapplicationslikeonlineemail, socialnetworksandonlineofficetoolssuchaswordprocessingandspread- sheets.Whilesuchwebapplicationsbecamemoreandmorecommonover thepastyears,onlyalimitednumberofwebapplicationbenchmarkingtools emerged.Hostingadvancedwebapplicationscanrequirelotsofresources.It isthereforeimportanttoperformresearchonhowtoimprovevariousaspects ([11],[12],[8],[16])ofsuchhostingsystems.Webapplicationbenchmarks areespeciallyimportantwhendoingsuchresearch,sincetheyprovideacon- figurable,reproducibleandoftenrealisticsimulationofrealwebapplication usage.Benchmarktoolsaidthesystematicresearchintotheperformanceof webhostingsystemsandmakeitpossibletocomparedifferentsystemsand differentsystemsetups. Thereareanumberofbenchmarkapplicationsthatareusedtoday,like TPC-W,RUBBoSandRUBiS.Thesebenchmarkshavesimilarcharacter- istics.TPC-Wsimulatesawebstore,whileRUBBoSisasimplebulletin boardsystemandRUBBiSmimicsanonlineauctionsite.Althoughthese toolshaveproventobeusefultomanyresearchers,theyhavelimitsinterms ofdatasetsize,scalabilityandfunctionality.Forexample,allthreetools runonasinglesystem.Thereisnobuilt-inwaytoscaleuptomultiplesys- tems.Althoughitcanbesubjecttoheateddebate,wefeelthatthesynthetic workloadsthesebenchmarkscreateareunrealisticandlackconfigurability. Anotherimportantdownsideofthesebenchmarksisthattheygeneratea constantloadthroughafixednumberofemulatedbrowsers.Theseemulated browserallwaitindefinitelyforaservertoanswerarequest,whileinreality avisitorisonlypreparedtowaitforalimitedamountoftime,like4to8 seconds[7].Inaddition,thenumberofvisitorstypicallyvariesgreatlyde- pendingonthetimeofday,whilemostbenchmarktoolsshowaconstant loadovertheentiretimeperiod.Sothesetoolslackrealism,flexibilityand configurability:qualitiesthatwethinkareveryimportantwhenitcomesto thedevelopmentandtestingofadvancedwebhostingsetups. Toaddresssomeoftheshortcomingsofthecurrentlyavailablebench- marktools,wecreatedWikiBench.WikiBenchhasanumberofadvantages comparedtothepreviouslydiscussedtools.Firstofall,Wikibenchoffersa highdegreeofrealism,sinceitisentirelybasedontheWikipediasoftware anddata.wehaveobtainedaccesstracesfromtheWikiMediaFoundation. ThesetracescontaindetailedtrafficlogsofrequestsmadetoWikipediabyits 2

users.Weareabletoconverttheseaccesstracestobenchmarkworkloadsby usingourspecializedTraceBenchtool.TraceBenchcanreducetheintensity ofthistrafficwhilemaintainingimportanttrafficproperties,allowingusto createveryrealisticbenchmarkworkloadswithintensitiesrangingfromvery lowuptotheoriginaltrafficintensityofthetracefile. Tomatchtheserversidesoftwarewiththeworkloadfiles,weusetheopen sourceMediaWikiapplication[1],thesoftwareusedtorunWikipedia.This applicationisquiteadvancedandhasbeentestedextensively.Inaddition wehaveusedpubliclyavailablesnapshotsfromtheWikiMediafoundationto buildamirroroftheEnglishWikipediasite.Sowenowhavearealworld webapplicationwithalargeamountofdatatoserve. SinceWikipediahasalargeandconstantlyincreasingamountofdata andvisitors,basingabenchmarktoolonthisdataisnotaneasytask.We havedesignedWikiBench,fromthegrounduptobeaninherentlydistributed application.Itcanscaleupfromonemachineforsmallbenchmarkstomany machinesworkingtogetherinacoordinatedfashion.Allputtogether,we believewehavecreatedabenchmarkingtoolthatisabletocreateaworkload whichmatchesrealityclosely.Byusingrealworldserversidesoftwareand data,wethinktheWikiBenchbenchmarkingsuiteisaveryrealisticand flexibleresearchtool. InitialbenchmarkresultsshowatypicaldayofWikipediatrafficand therelationbetweentherequestrateandtheserverresponsetimes.The intensityofthistrafficisreducedwithourTraceBenchtooltofitoursystem undertest. Therestofthisthesisisorganizedasfollows.InSection2wediscussa numberofexistingwebapplicationbenchmarks,inSection3wedescribethe WikiBenchdesignindetail.InSection4wefocusonhowwecreaterealistic workloadsofarbitrarysizefromthetracefiles.Section5discussesourinitial resultsandsection6concludes. 3

2RelatedWork Oneofthemorewellknownandintensivelyusedtoolstobenchmarkapplica- tionhostingsystemsisTPCBenchmarkW TM [13].Anothertoolwewilldis- cuss,intendedforbenchmarkingcachingservers,iscalledWebPolygraph[10]. Althoughnotaimedatapplicationbenchmarking,itdoeshaveanumberof interestingfeatures. 2.1TPC-W TPCBenchmarkW TM (TPC-W)isatransactionalwebbenchmark[13].The workloadisperformedinacontrolledinternetcommerceenvironmentthat simulatestheactivitiesofabusinessorientedtransactionalwebserver.This isdonebyimplementingawebstoreandatestingtoolthatbrowsesthrough thisstore.TPC-WusestheconceptofEmulatedBrowsers(EBs).Each EBrunsinaseparatethreadandemulatesauser’sbrowserandtheuser actions.TheEBusesMarkovchainstofindrandompathsintheTPC-W store.AnEBhasarandomthinktimethatemulatesthetimeausertakes beforeclickingonthenextlink.Thisthinktimeisdistributedinsucha waythattheaveragethinktimeis7seconds.LoadgenerationinTPC- Whappensinabest-effortfashion.ThismeansthattheEBswaitforeach requesttobeprocessed,nomatterhowlongthistakes.Theresultbeingthat thisbenchmarktoolcanonlyvaryloadbychangingthedegreeofconcurrent emulatedbrowsers.insteadofchangingtherequestrate,whichwouldmatch realitymuchcloser. TPC-Woffersthreedifferentmodes.Thesemodeshavedifferentratios betweentheamountofinformationrequests(pagereads)andthenumberof ordersplacesbycustomers.Inthedefaultmode,theratioofreadandorder pagesis95%/5%.Inbrowsemode,usersmostlybrowsearoundwith98% browsingand2%ordering.Inordermode,usersmostlybuystuffandorder pagesarevisited50%ofthetime.Ordermodeputsmuchmorepressureon thedatabasesystemofaTPC-Wsite,sincetheactionsassociatedwiththe orderingofproductsarenotcacheable.Sothelattermodecanbeusedto putmorepressureonthedatabasewithouthavingtoincreasethewebserver capacity.TPC-WBenchmarkhasbeendiscontinuedsince2005,butisstill usedfortestingdistributedhostingsystems. TwoothercommonlyusedbenchmarktoolsarecalledRUBBoS[2]and RUBiS[3].BothhavesimilarcharacteristicstoTPC-W.RUBBoSisabul- letinboardbenchmarkmodeledafteranonlinenewsforumlikeSlashdot. RUBiSisabenchmarkingtoolbasedonanonlineauctionsitelikeeBay. BothRUBBoSandRUBiSmakeuseofemulatedusersessions,transition 4

tableswithprobabilitiesandwaittimes. Althoughwebsitetrafficdoesoftenhavereturningandpredictablepat- terns,wefeelthattheworkloadsofthesetoolsaretoolimited.Alarger problemweseeisthattheemulatedbrowserscreateaverystableandpre- dictableworkloadonasystem.Thesetools,forexample,donotcreatespikes oftrafficatthepagelevel,whileinrealitypagesgetlinkedonlargenews forumslikeDiggorSlashdot.Wecallthesetrafficspikesflashcrowds.A singlepagemay,forshortperiodsoftime,easilyreceivemanytimesmore trafficthanitgetsonaverage.Foradistributedhostingsystemitiseasy toanticipatethestabletrafficpatternandpagepopularitydistributionthat Markovchainscreate.Butinrealityadistributedhostingsystemwillhave todealwithmoredifficulttrafficpatternslikethepagelevelflashcrowds describedabove.Suchflashcrowds,evensmallones,changethespatialand temporallocalityoftherequests,possiblyoverloadingpartsofthesystemif notimelymeasuresaretaken. Anotherproblemweseewiththeemulatedbrowsersinthedescribed benchmarkingtoolsisthattheyuseafixedamountofconcurrentemulated browserstoputloadonthesystemundertest.Sotheloadputonahost- ingsystemisdefinedbytheamountofemulatedbrowsersinsteadofby therequestrate.Iftheserverisoverloaded,theemulatedbrowserswillall waitlonger,decreasingtheloadputontheserveratsomepoint.Workload andresponsetimearethereforedirectlyrelatedtoeachother.Thisishow- evernotrepresentativeofrealworldtraffic,wherenewandcurrentvisitors willkeephammeringahostingsystemwithnewrequests.Inotherwords, thesebenchmarksdefineserverloadbythenumberofconcurrentemulated browsers,insteadofbythenumberofrequestspertimeunitaserveractually canhandlebeforeitstartstoreactunacceptablyslow. Allthreesynthetictrafficgeneratorsalsolackthetrafficthatisgenerated bywebcrawlers,malicioususersandpagescrapers.Suchuserscanforex- amplehaveaveryconstantthinktimeanddifferentpagepreferencesthana regularvisitor.Acrawlerwillgenerallyvisitallpagesinasite,whilemost regularuserswillonlyvisitasmallsubset.Amalicioususermightsubmit loadsofdifficultsearchqueriesorextraordinaryamountsofpageeditsto vandalizethecontent.Pagescrapersmightonlydownloadthepageandnot therelateditems,likeimagesandstylesheets.Theseusersexistinthereal world,soasystemshouldbeabletodealwiththem.Onalargescalehost- ingsystemlikethatofWikipedia,allthiscontributesonlyslightlytothe trafficsoitisnotunreasonabletonottakethisintoaccountinasynthetic workload. Finally,allthreebenchmarksarenotdesignedtoscale.Oneneedsto startmultiplemachinesbyhandandcreatetoolstosynchronizethestartof 5

theseindividualmachines.Theseindividualbenchmarkprocesseswillnot coordinatewitheachother,makingitmoredifficulttodetecterrors.E.g. ifonefails,othermachineswillkeepgeneratingloadwithoutnoticingthis failure. 2.2WebPolygraph WebPolygraph[10]isatoolforbenchmarkingHTTPintermediarieslike proxyserversandotherwebcachingproducts.Eventhoughthisbenchmark testsHTTPintermediariesinsteadofHTTPservers,WebPolygraphisstill interestingenoughtomentionsincetheauthorshavetakensomuchcareto createrealisticworkloads.TheWebPolygraphbenchmarkisbasedentirely onsynthetictrafficworkloads,whicharecreatedinconjunctionwithindustry andvariousresearchgroups.ThecreatorsofWebPolygraphhavetriedto createrealisticworkloadsbyanalyzingandusingmanycharacteristicsofreal webtraffic,likefiletypeandsizedistributionandrequestinterarrivaltimes. Fromtheirresearch,itbecomesclearthatcreationofsynthetic,realistic workloadisaresearchfieldinitself. TheWebPolygraphglobalarchitectureconsistsofvirtualclientsand servers.Clientsrequestsimulatedobjectsfromtheseservers.TheHTTP intermediariesareplacedinbetween.Serversandclientsaregluedtogether byusingconfigurationfilesthataresharedbetweenthetwo. Thechoiceforsyntheticworkloadsismadebecauseforthistool’spur- pose,realaccesstraceswouldneedtobemodifiedheavily.Forexample,the benchmarkallowschangestoparameterssuchastherequestrate,cachehit ratiosandfilesizeandpopularitydistribution.Thecreatorsarguethatusing different(realworld)tracestogetdifferenttypesofworkloadswillchange manyparametersatonceinsteadofjustoneoftheparameters.Thismakes analyzingperformancecomparisonsmoredifficult,sincedifferentworkloads arenotascloselyrelatedtoeachotherasispossiblewithsyntheticallycre- atedworkloads.Itisalsoarguedthatmanytestsdonotcorrespondtoany realtrace,sousingrealtraceswouldmakenosense.Furthermore,usingreal tracesforthisbenchmarkwouldmeanthatthebenchmarkneedstohave detailedknowledgeofmillionsofobjects.Instead,theauthorshavechosen toembedinformationabouttheobjectsrequestedfromtheserverinthe URLsofthoseobjects.Forexample,anobjecttypeidembeddedintheurl identifiesthepropertieslikefilesizeandfiletypeoftherequestedobject. Tovarytheworkload,theauthorscreatedamodelthatspecifiesamean inter-arrivaltimeforaPoissonstreamofrequests.Therobots,whichare similartothepreviouslymentionedemulatedbrowsers,multiplythismean bythecurrentloadfactortogetthecorrectinter-arrivaltimeatanymoment. 6

Unfortunately,thisbenchmarktoolisoflittleusewhenitcomestotest- ingsystemsthathostwebapplications.Becauseitisaimedatbenchmarking cachingservers,theserversideneedstohostaveryspecificwebapplication andwebserverthatgeneratesdatafilesofdifferentsizeandtypebasedon thefilename.Thesedatafilescontainrandomdata.Thebenchmarkingap- plicationrequestsfilesofcertainsizesinacertaindistributionbyadjusting therequestedfilenames.Unfortunately,thewebapplicationisnotrepresen- tativeofatypicalwebapplication.E.g.itdoesnotservewebpagesthatlink toeachother.Inaddition,itdoesnotdependonadatabase,makingiteven lessrealisticsincemostadvancedwebapplicationsareverydataintensive. 7

3SystemModel WikiBenchconsistsofacollectionoftools.Themaintoolsare: • TraceBench:AtooltoprocesstracefilesforusewithWikiBench • WikiBench:thebenchmarkapplicationitself,whichcanbedivided intoacontrollerandworkernodes • Post-processingtools:Asetofscriptstoprocesslogfilesproduced byWikiBenchandcreategraphicsfromtheresults WikiBenchisinthefirstplacecreatedasaresearchtool.Ourgoalisto createarealisticbenchmarkwithadaptabletrafficproperties.Wetherefore benchmarkarealworldsoftwareapplicationthatisusedextensively.This softwareapplicationisMediaWiki,thesoftwarethatisusedbyWikipedia. Wikipediaisacollaborative,multi-languageonlineencyclopedia.Wikipedia isbasedonwikitechnologyforstoringandgivingstructuretoinformation. TheMediaWikiapplicationrepresentstherealitieswearefacedwithtoday whendesigningsoftwareforalargescalewebsitewithmanyusers.Thisin turnmeansitwillputrealistic,highdemandsonahostingsystemthatisused tohostthiswebapplication.AnaddedadvantagetousingWikipediaisthat themillionsofWikipediapagesarefreelyavailableintheformofsnapshots [4].SousingMediaWikiinconjunctionwithWikipediasnapshotsallowsus tocreatearealworldapplicationwithrealworlddatatobebenchmarked. Aswasnotedbyotherauthors,likethoseofWebPolygraph[10],creating usefulworkloadswithabenchmarkingtoolisaresearchfieldinitself.We areverygratefultohaverealaccesstracesfromWikipedia.Thesetraces,ob- taineddirectlyfromtheWikiMediafoundation,arecompletelyanonymized. Foreachrequest,weonlyhaveaunixtimestamp,thecompleteurlthatwas accessedanda’save’flagthatissettotrueonlyiftherequestresultedin apagechangeorpagecreation.Nomorepersonalinformationthanavail- ableonthepubliclyaccessibleWikipediawebsitecanbedeductedfromthese traces.E.g.wedonothaveaccesstoIPaddresses,cookies,useraccount informationorthecontentsofaPOSTrequest.Ontopofthat,eachrequest onlyhasa10%chanceofbeingincludedinthetracesweget.Previousre- search[15]hasshownthatthesetracescontainarepresentativesampleof theworkloadWikipediagets.Wethereforeusedthesetracestocreatework- loadsforWikiBenchinsteadofcreatingpurelysyntheticworkloadslikeother benchmarkingtoolshavedone.Thisaddsrealismtothebenchmark,since thetracesweusecontainallthecharacteristicsthattherealWikipediasite hastodealwith. 8

Inthisthesiswedefinethesystemundertest(SUT)asthehostingsystem thathoststheWikipediadata.WeapproachthisSUTasablackbox,with aURLthatservesasourentrypoint.Itisoutofthescopeofthisthesisto definedetailedrequirementsonahostingsystem.Inprinciple,thehosting systemcanapplyanytechniquethatonecanthinkoftoimproveoverall systemperformance.Theonly“common-sense”requirementsweputonthe systemisthatisisaccessiblethroughHTTP,exactlyliketheMediaWiki software,andthatitgivesthesameresponsesinidenticalsituations.Ifthe SUTwouldberunsidebysidewithavanillaMediaWikiinstallation,we expecttheexactsamereplieswhendoingidenticalGETandPOSTrequests onbothservers.Thereisoneexceptiontothisrule:weremovedacheck forduplicateeditsandCSRFattacks([5],[17])fromMediaWiki.Weworked aroundthischeckbycommentingoutasinglelineofMediaWikicode.With- outthischeck,wecanpostanythingwewanttoawikipagewithasinglehttp POSTrequest.Ifwewouldleavethissecuritycheckinthecode,wewould needtofirstrequesttheeditpage,parsethatpagetoobtainanumberof hiddenformelementsandthenperformthePOSTrequest.Thehiddenfield, asecrettokenthatisuserspecific,isusedtopreventCSRFattacksinwhich aloggedonusermightPOSTdatatoWikipediawithoutknowingitwhile heisvisitingamaliciouswebsite.Becauseamalicioussitecannotguess orcalculatethissecrettoken,itcanneverperformavalidPOSTrequest withouttheusernoticingit.Leavingthecheckinplacewouldalsoincrease thechanceofaneditconflict.Whilerequestingandparsingtheeditpage,a timewindowisintroducedinwhichanotherthreadmightrequestthatsame page.BecauseMediaWikiusestimestampstocheckforeditconflicts,we willnowhavetosolveaneditconflictbetweentwothreads,introducingeven morecomplexity. 3.1Requirements Wikipediahasanumberofcharacteristicsthatmakeitidealtobeused inadistributedwebapplicationbenchmark.Firstofall,thereislotsof data.JusttheEnglishwiki,withoutuserandtalkpages,containsmore than7millionarticlesinthesnapshottakenonOctober7,2008.Thiscan behostedononesingleserver,butthatserverwillonlybeabletosustain fractionsofWikipedia’sactualtraffic.Wikipediareceiveshighnumbersof requestspersecond.Asofthiswriting,peaksof50,000to60,000requestsper secondarenoexception.So,evenwithour10%sampleoftheaccesstraces, wecangeneratehighloadsuptoabout5000requestsperseconds.The requirementsweputonWikiBenchobviouslyneedtomatchthesenumbers. Thissubsectiondescribesthemaingoalsandrequirementswehaveputon 9