WikiBench: A distributed, Wikipedia based web application ...

WikiBench: A distributed, Wikipedia based web application ...

Documents
33 pages
Lire
Le téléchargement nécessite un accès à la bibliothèque YouScribe
Tout savoir sur nos offres

Description

WikiBench: A distributed, Wikipedia
based web application benchmark
Master thesis by Erik-Jan van Baaren
Student number 1278967
erikjan@gmail.com
Under the supervision of:
Guillaume Pierre
Guido Urdaneta
Vrije Univesiteit Amsterdam
Department of Computer Science
May 13, 2009 Abstract
Many di erent, novel approaches have been taken to improve throughput
and scalability of distributed web application hosting systems and relational
databases. Yet there are only a limited number of web application bench-
marks available. We present the design and implementation of WikiBench,
a distributed web application benchmarking tool based on Wikipedia. Wik-
iBench is a trace based benchmark, able to create realistic workloads with
thousands of requests per second to any system hosting the freely available
Wikipedia data and software. We obtained completely anonymized, sam-
pled access traces from the Wikimedia Foundation, and we created software
to process these traces in order to reduce the intensity of its tra c while
still maintaining the most important properties such as inter-arrival times
and distribution of page popularity. This makes WikiBench usable for both
small and large scale benchmarks. Initial benchmarks show a regular day of
tra c with its ups and downs. By using median response times, we are able
to show the e ects of increasing tra c intensities on our system under test. Contents
1 Introduction 2
2 Related Work 4
2.1 TPC-W . . . . . . . . . . . . . . . . . . . . . . . ...

Sujets

Informations

Publié par
Nombre de visites sur la page 148
Langue English
Signaler un problème
WbiakisBedenwceh:baApdpilisctraitbioutnedb,enWcihkmipaerdkiaMasterthesisbyErik-JanvanBaarenStudentnumber1278967erikjan@gmail.comUnderthesupervisionof:GuillaumePierreGuidoUrdanetaVrijeUnivesiteitAmsterdamDepartmentofComputerScienceMay13,2009
AbstractManydifferent,novelapproacheshavebeentakentoimprovethroughputandscalabilityofdistributedwebapplicationhostingsystemsandrelationaldatabases.Yetthereareonlyalimitednumberofwebapplicationbench-marksavailable.WepresentthedesignandimplementationofWikiBench,adistributedwebapplicationbenchmarkingtoolbasedonWikipedia.Wik-iBenchisatracebasedbenchmark,abletocreaterealisticworkloadswiththousandsofrequestspersecondtoanysystemhostingthefreelyavailableWikipediadataandsoftware.Weobtainedcompletelyanonymized,sam-pledaccesstracesfromtheWikimediaFoundation,andwecreatedsoftwaretoprocessthesetracesinordertoreducetheintensityofitstrafficwhilestillmaintainingthemostimportantpropertiessuchasinter-arrivaltimesanddistributionofpagepopularity.ThismakesWikiBenchusableforbothsmallandlargescalebenchmarks.Initialbenchmarksshowaregulardayoftrafficwithitsupsanddowns.Byusingmedianresponsetimes,weareabletoshowtheeffectsofincreasingtrafficintensitiesonoursystemundertest.
Contents1Introduction22RelatedWork42.1TPC-W..............................42.2WebPolygraph..........................63SystemModel83.1Requirements...........................93.2WikiBenchdesign.........................113.3TraceBenchDesign........................153.4WikiBenchWorkow.......................164WorkloadCreation194.1ChangingtheRequestRate...................194.2AHybridApproach........................215BenchmarkResults326FutureWork276.1ScalinguptracandFlashCrowds...............276.2Adjustread/writeratio......................276.3Adjustthedistributionofpagepopularity...........276.4Indicationofrealism.......................286.5Moreadvancededits.......................287Conclusion192
1IntroductionAlthoughoriginallyaplacedesignedforresearcherstoeasilyexchangein-formation,theworldwidewebhasbecomeoneofthemostimportantinfor-mationinfrastructuresofmodernsociety.WehaveseenarapidtransitionfromstaticHTMLdocumentstoadvancedwebapplicationslikeonlineemail,socialnetworksandonlineofficetoolssuchaswordprocessingandspread-sheets.Whilesuchwebapplicationsbecamemoreandmorecommonoverthepastyears,onlyalimitednumberofwebapplicationbenchmarkingtoolsemerged.Hostingadvancedwebapplicationscanrequirelotsofresources.Itisthereforeimportanttoperformresearchonhowtoimprovevariousaspects([11],[12],[8],[16])ofsuchhostingsystems.Webapplicationbenchmarksareespeciallyimportantwhendoingsuchresearch,sincetheyprovideacon-figurable,reproducibleandoftenrealisticsimulationofrealwebapplicationusage.Benchmarktoolsaidthesystematicresearchintotheperformanceofwebhostingsystemsandmakeitpossibletocomparedifferentsystemsanddifferentsystemsetups.Thereareanumberofbenchmarkapplicationsthatareusedtoday,likeTPC-W,RUBBoSandRUBiS.Thesebenchmarkshavesimilarcharacter-istics.TPC-Wsimulatesawebstore,whileRUBBoSisasimplebulletinboardsystemandRUBBiSmimicsanonlineauctionsite.Althoughthesetoolshaveproventobeusefultomanyresearchers,theyhavelimitsintermsofdatasetsize,scalabilityandfunctionality.Forexample,allthreetoolsrunonasinglesystem.Thereisnobuilt-inwaytoscaleuptomultiplesys-tems.Althoughitcanbesubjecttoheateddebate,wefeelthatthesyntheticworkloadsthesebenchmarkscreateareunrealisticandlackconfigurability.Anotherimportantdownsideofthesebenchmarksisthattheygenerateaconstantloadthroughafixednumberofemulatedbrowsers.Theseemulatedbrowserallwaitindefinitelyforaservertoanswerarequest,whileinrealityavisitorisonlypreparedtowaitforalimitedamountoftime,like4to8seconds[7].Inaddition,thenumberofvisitorstypicallyvariesgreatlyde-pendingonthetimeofday,whilemostbenchmarktoolsshowaconstantloadovertheentiretimeperiod.Sothesetoolslackrealism,flexibilityandconfigurability:qualitiesthatwethinkareveryimportantwhenitcomestothedevelopmentandtestingofadvancedwebhostingsetups.Toaddresssomeoftheshortcomingsofthecurrentlyavailablebench-marktools,wecreatedWikiBench.WikiBenchhasanumberofadvantagescomparedtothepreviouslydiscussedtools.Firstofall,Wikibenchoffersahighdegreeofrealism,sinceitisentirelybasedontheWikipediasoftwareanddata.wehaveobtainedaccesstracesfromtheWikiMediaFoundation.ThesetracescontaindetailedtrafficlogsofrequestsmadetoWikipediabyits2
users.WeareabletoconverttheseaccesstracestobenchmarkworkloadsbyusingourspecializedTraceBenchtool.TraceBenchcanreducetheintensityofthistrafficwhilemaintainingimportanttrafficproperties,allowingustocreateveryrealisticbenchmarkworkloadswithintensitiesrangingfromverylowuptotheoriginaltrafficintensityofthetracefile.Tomatchtheserversidesoftwarewiththeworkloadfiles,weusetheopensourceMediaWikiapplication[1],thesoftwareusedtorunWikipedia.Thisapplicationisquiteadvancedandhasbeentestedextensively.InadditionwehaveusedpubliclyavailablesnapshotsfromtheWikiMediafoundationtobuildamirroroftheEnglishWikipediasite.Sowenowhavearealworldwebapplicationwithalargeamountofdatatoserve.SinceWikipediahasalargeandconstantlyincreasingamountofdataandvisitors,basingabenchmarktoolonthisdataisnotaneasytask.WehavedesignedWikiBench,fromthegrounduptobeaninherentlydistributedapplication.Itcanscaleupfromonemachineforsmallbenchmarkstomanymachinesworkingtogetherinacoordinatedfashion.Allputtogether,webelievewehavecreatedabenchmarkingtoolthatisabletocreateaworkloadwhichmatchesrealityclosely.Byusingrealworldserversidesoftwareanddata,wethinktheWikiBenchbenchmarkingsuiteisaveryrealisticandflexibleresearchtool.InitialbenchmarkresultsshowatypicaldayofWikipediatrafficandtherelationbetweentherequestrateandtheserverresponsetimes.TheintensityofthistrafficisreducedwithourTraceBenchtooltofitoursystemundertest.Therestofthisthesisisorganizedasfollows.InSection2wediscussanumberofexistingwebapplicationbenchmarks,inSection3wedescribetheWikiBenchdesignindetail.InSection4wefocusonhowwecreaterealisticworkloadsofarbitrarysizefromthetracefiles.Section5discussesourinitialresultsandsection6concludes.3
2RelatedWorkOneofthemorewellknownandintensivelyusedtoolstobenchmarkapplica-tionhostingsystemsisTPCBenchmarkWTM[13].Anothertoolwewilldis-cuss,intendedforbenchmarkingcachingservers,iscalledWebPolygraph[10].Althoughnotaimedatapplicationbenchmarking,itdoeshaveanumberofinterestingfeatures.2.1TPC-WTPCBenchmarkWTM(TPC-W)isatransactionalwebbenchmark[13].Theworkloadisperformedinacontrolledinternetcommerceenvironmentthatsimulatestheactivitiesofabusinessorientedtransactionalwebserver.Thisisdonebyimplementingawebstoreandatestingtoolthatbrowsesthroughthisstore.TPC-WusestheconceptofEmulatedBrowsers(EBs).EachEBrunsinaseparatethreadandemulatesauser’sbrowserandtheuseractions.TheEBusesMarkovchainstofindrandompathsintheTPC-Wstore.AnEBhasarandomthinktimethatemulatesthetimeausertakesbeforeclickingonthenextlink.Thisthinktimeisdistributedinsuchawaythattheaveragethinktimeis7seconds.LoadgenerationinTPC-Whappensinabest-effortfashion.ThismeansthattheEBswaitforeachrequesttobeprocessed,nomatterhowlongthistakes.Theresultbeingthatthisbenchmarktoolcanonlyvaryloadbychangingthedegreeofconcurrentemulatedbrowsers.insteadofchangingtherequestrate,whichwouldmatchrealitymuchcloser.TPC-Woffersthreedifferentmodes.Thesemodeshavedifferentratiosbetweentheamountofinformationrequests(pagereads)andthenumberofordersplacesbycustomers.Inthedefaultmode,theratioofreadandorderpagesis95%/5%.Inbrowsemode,usersmostlybrowsearoundwith98%browsingand2%ordering.Inordermode,usersmostlybuystuffandorderpagesarevisited50%ofthetime.OrdermodeputsmuchmorepressureonthedatabasesystemofaTPC-Wsite,sincetheactionsassociatedwiththeorderingofproductsarenotcacheable.Sothelattermodecanbeusedtoputmorepressureonthedatabasewithouthavingtoincreasethewebservercapacity.TPC-WBenchmarkhasbeendiscontinuedsince2005,butisstillusedfortestingdistributedhostingsystems.TwoothercommonlyusedbenchmarktoolsarecalledRUBBoS[2]andRUBiS[3].BothhavesimilarcharacteristicstoTPC-W.RUBBoSisabul-letinboardbenchmarkmodeledafteranonlinenewsforumlikeSlashdot.RUBiSisabenchmarkingtoolbasedonanonlineauctionsitelikeeBay.BothRUBBoSandRUBiSmakeuseofemulatedusersessions,transition4
tableswithprobabilitiesandwaittimes.Althoughwebsitetrafficdoesoftenhavereturningandpredictablepat-terns,wefeelthattheworkloadsofthesetoolsaretoolimited.Alargerproblemweseeisthattheemulatedbrowserscreateaverystableandpre-dictableworkloadonasystem.Thesetools,forexample,donotcreatespikesoftrafficatthepagelevel,whileinrealitypagesgetlinkedonlargenewsforumslikeDiggorSlashdot.Wecallthesetrafficspikesflashcrowds.Asinglepagemay,forshortperiodsoftime,easilyreceivemanytimesmoretrafficthanitgetsonaverage.ForadistributedhostingsystemitiseasytoanticipatethestabletrafficpatternandpagepopularitydistributionthatMarkovchainscreate.Butinrealityadistributedhostingsystemwillhavetodealwithmoredifficulttrafficpatternslikethepagelevelflashcrowdsdescribedabove.Suchflashcrowds,evensmallones,changethespatialandtemporallocalityoftherequests,possiblyoverloadingpartsofthesystemifnotimelymeasuresaretaken.Anotherproblemweseewiththeemulatedbrowsersinthedescribedbenchmarkingtoolsisthattheyuseafixedamountofconcurrentemulatedbrowserstoputloadonthesystemundertest.Sotheloadputonahost-ingsystemisdefinedbytheamountofemulatedbrowsersinsteadofbytherequestrate.Iftheserverisoverloaded,theemulatedbrowserswillallwaitlonger,decreasingtheloadputontheserveratsomepoint.Workloadandresponsetimearethereforedirectlyrelatedtoeachother.Thisishow-evernotrepresentativeofrealworldtraffic,wherenewandcurrentvisitorswillkeephammeringahostingsystemwithnewrequests.Inotherwords,thesebenchmarksdefineserverloadbythenumberofconcurrentemulatedbrowsers,insteadofbythenumberofrequestspertimeunitaserveractuallycanhandlebeforeitstartstoreactunacceptablyslow.Allthreesynthetictrafficgeneratorsalsolackthetrafficthatisgeneratedbywebcrawlers,malicioususersandpagescrapers.Suchuserscanforex-amplehaveaveryconstantthinktimeanddifferentpagepreferencesthanaregularvisitor.Acrawlerwillgenerallyvisitallpagesinasite,whilemostregularuserswillonlyvisitasmallsubset.Amalicioususermightsubmitloadsofdifficultsearchqueriesorextraordinaryamountsofpageeditstovandalizethecontent.Pagescrapersmightonlydownloadthepageandnottherelateditems,likeimagesandstylesheets.Theseusersexistintherealworld,soasystemshouldbeabletodealwiththem.Onalargescalehost-ingsystemlikethatofWikipedia,allthiscontributesonlyslightlytothetrafficsoitisnotunreasonabletonottakethisintoaccountinasyntheticworkload.Finally,allthreebenchmarksarenotdesignedtoscale.Oneneedstostartmultiplemachinesbyhandandcreatetoolstosynchronizethestartof5
theseindividualmachines.Theseindividualbenchmarkprocesseswillnotcoordinatewitheachother,makingitmoredifficulttodetecterrors.E.g.ifonefails,othermachineswillkeepgeneratingloadwithoutnoticingthisfailure.2.2WebPolygraphWebPolygraph[10]isatoolforbenchmarkingHTTPintermediarieslikeproxyserversandotherwebcachingproducts.EventhoughthisbenchmarktestsHTTPintermediariesinsteadofHTTPservers,WebPolygraphisstillinterestingenoughtomentionsincetheauthorshavetakensomuchcaretocreaterealisticworkloads.TheWebPolygraphbenchmarkisbasedentirelyonsynthetictrafficworkloads,whicharecreatedinconjunctionwithindustryandvariousresearchgroups.ThecreatorsofWebPolygraphhavetriedtocreaterealisticworkloadsbyanalyzingandusingmanycharacteristicsofrealwebtraffic,likefiletypeandsizedistributionandrequestinterarrivaltimes.Fromtheirresearch,itbecomesclearthatcreationofsynthetic,realisticworkloadisaresearchfieldinitself.TheWebPolygraphglobalarchitectureconsistsofvirtualclientsandservers.Clientsrequestsimulatedobjectsfromtheseservers.TheHTTPintermediariesareplacedinbetween.Serversandclientsaregluedtogetherbyusingconfigurationfilesthataresharedbetweenthetwo.Thechoiceforsyntheticworkloadsismadebecauseforthistool’spur-pose,realaccesstraceswouldneedtobemodifiedheavily.Forexample,thebenchmarkallowschangestoparameterssuchastherequestrate,cachehitratiosandfilesizeandpopularitydistribution.Thecreatorsarguethatusingdifferent(realworld)tracestogetdifferenttypesofworkloadswillchangemanyparametersatonceinsteadofjustoneoftheparameters.Thismakesanalyzingperformancecomparisonsmoredifficult,sincedifferentworkloadsarenotascloselyrelatedtoeachotherasispossiblewithsyntheticallycre-atedworkloads.Itisalsoarguedthatmanytestsdonotcorrespondtoanyrealtrace,sousingrealtraceswouldmakenosense.Furthermore,usingrealtracesforthisbenchmarkwouldmeanthatthebenchmarkneedstohavedetailedknowledgeofmillionsofobjects.Instead,theauthorshavechosentoembedinformationabouttheobjectsrequestedfromtheserverintheURLsofthoseobjects.Forexample,anobjecttypeidembeddedintheurlidentifiesthepropertieslikefilesizeandfiletypeoftherequestedobject.Tovarytheworkload,theauthorscreatedamodelthatspecifiesameaninter-arrivaltimeforaPoissonstreamofrequests.Therobots,whicharesimilartothepreviouslymentionedemulatedbrowsers,multiplythismeanbythecurrentloadfactortogetthecorrectinter-arrivaltimeatanymoment.6
Unfortunately,thisbenchmarktoolisoflittleusewhenitcomestotest-ingsystemsthathostwebapplications.Becauseitisaimedatbenchmarkingcachingservers,theserversideneedstohostaveryspecificwebapplicationandwebserverthatgeneratesdatafilesofdifferentsizeandtypebasedonthefilename.Thesedatafilescontainrandomdata.Thebenchmarkingap-plicationrequestsfilesofcertainsizesinacertaindistributionbyadjustingtherequestedfilenames.Unfortunately,thewebapplicationisnotrepresen-tativeofatypicalwebapplication.E.g.itdoesnotservewebpagesthatlinktoeachother.Inaddition,itdoesnotdependonadatabase,makingitevenlessrealisticsincemostadvancedwebapplicationsareverydataintensive.7
3SystemModelWikiBenchconsistsofacollectionoftools.Themaintoolsare:TraceBench:AtooltoprocesstracefilesforusewithWikiBenchWikiBench:thebenchmarkapplicationitself,whichcanbedividedintoacontrollerandworkernodesPost-processingtools:AsetofscriptstoprocesslogfilesproducedbyWikiBenchandcreategraphicsfromtheresultsWikiBenchisinthefirstplacecreatedasaresearchtool.Ourgoalistocreatearealisticbenchmarkwithadaptabletrafficproperties.Wethereforebenchmarkarealworldsoftwareapplicationthatisusedextensively.ThissoftwareapplicationisMediaWiki,thesoftwarethatisusedbyWikipedia.Wikipediaisacollaborative,multi-languageonlineencyclopedia.Wikipediaisbasedonwikitechnologyforstoringandgivingstructuretoinformation.TheMediaWikiapplicationrepresentstherealitieswearefacedwithtodaywhendesigningsoftwareforalargescalewebsitewithmanyusers.Thisinturnmeansitwillputrealistic,highdemandsonahostingsystemthatisusedtohostthiswebapplication.AnaddedadvantagetousingWikipediaisthatthemillionsofWikipediapagesarefreelyavailableintheformofsnapshots[4].SousingMediaWikiinconjunctionwithWikipediasnapshotsallowsustocreatearealworldapplicationwithrealworlddatatobebenchmarked.Aswasnotedbyotherauthors,likethoseofWebPolygraph[10],creatingusefulworkloadswithabenchmarkingtoolisaresearchfieldinitself.WeareverygratefultohaverealaccesstracesfromWikipedia.Thesetraces,ob-taineddirectlyfromtheWikiMediafoundation,arecompletelyanonymized.Foreachrequest,weonlyhaveaunixtimestamp,thecompleteurlthatwasaccessedanda’save’flagthatissettotrueonlyiftherequestresultedinapagechangeorpagecreation.Nomorepersonalinformationthanavail-ableonthepubliclyaccessibleWikipediawebsitecanbedeductedfromthesetraces.E.g.wedonothaveaccesstoIPaddresses,cookies,useraccountinformationorthecontentsofaPOSTrequest.Ontopofthat,eachrequestonlyhasa10%chanceofbeingincludedinthetracesweget.Previousre-search[15]hasshownthatthesetracescontainarepresentativesampleoftheworkloadWikipediagets.Wethereforeusedthesetracestocreatework-loadsforWikiBenchinsteadofcreatingpurelysyntheticworkloadslikeotherbenchmarkingtoolshavedone.Thisaddsrealismtothebenchmark,sincethetracesweusecontainallthecharacteristicsthattherealWikipediasitehastodealwith.8
Inthisthesiswedefinethesystemundertest(SUT)asthehostingsystemthathoststheWikipediadata.WeapproachthisSUTasablackbox,withaURLthatservesasourentrypoint.Itisoutofthescopeofthisthesistodefinedetailedrequirementsonahostingsystem.Inprinciple,thehostingsystemcanapplyanytechniquethatonecanthinkoftoimproveoverallsystemperformance.Theonly“common-sense”requirementsweputonthesystemisthatisisaccessiblethroughHTTP,exactlyliketheMediaWikisoftware,andthatitgivesthesameresponsesinidenticalsituations.IftheSUTwouldberunsidebysidewithavanillaMediaWikiinstallation,weexpecttheexactsamereplieswhendoingidenticalGETandPOSTrequestsonbothservers.Thereisoneexceptiontothisrule:weremovedacheckforduplicateeditsandCSRFattacks([5],[17])fromMediaWiki.WeworkedaroundthischeckbycommentingoutasinglelineofMediaWikicode.With-outthischeck,wecanpostanythingwewanttoawikipagewithasinglehttpPOSTrequest.Ifwewouldleavethissecuritycheckinthecode,wewouldneedtofirstrequesttheeditpage,parsethatpagetoobtainanumberofhiddenformelementsandthenperformthePOSTrequest.Thehiddenfield,asecrettokenthatisuserspecific,isusedtopreventCSRFattacksinwhichaloggedonusermightPOSTdatatoWikipediawithoutknowingitwhileheisvisitingamaliciouswebsite.Becauseamalicioussitecannotguessorcalculatethissecrettoken,itcanneverperformavalidPOSTrequestwithouttheusernoticingit.Leavingthecheckinplacewouldalsoincreasethechanceofaneditconflict.Whilerequestingandparsingtheeditpage,atimewindowisintroducedinwhichanotherthreadmightrequestthatsamepage.BecauseMediaWikiusestimestampstocheckforeditconflicts,wewillnowhavetosolveaneditconflictbetweentwothreads,introducingevenmorecomplexity.3.1RequirementsWikipediahasanumberofcharacteristicsthatmakeitidealtobeusedinadistributedwebapplicationbenchmark.Firstofall,thereislotsofdata.JusttheEnglishwiki,withoutuserandtalkpages,containsmorethan7millionarticlesinthesnapshottakenonOctober7,2008.Thiscanbehostedononesingleserver,butthatserverwillonlybeabletosustainfractionsofWikipedia’sactualtraffic.Wikipediareceiveshighnumbersofrequestspersecond.Asofthiswriting,peaksof50,000to60,000requestspersecondarenoexception.So,evenwithour10%sampleoftheaccesstraces,wecangeneratehighloadsuptoabout5000requestsperseconds.TherequirementsweputonWikiBenchobviouslyneedtomatchthesenumbers.Thissubsectiondescribesthemaingoalsandrequirementswehaveputon9