Towards a User-Oriented Benchmark for Transport Protocols ...

Ludovic Hablot

Description

Laboratoire de l’Informatique du Parallélisme
École Normale Supérieure de Lyon
oUnité Mixte de Recherche CNRS-INRIA-ENS LYON-UCBL n 5668
Towards a User-Oriented Benchmark for
Transport Protocols Comparison in very High
Speed Networks
Romaric Guillier ,
July 2007Ludovic Hablot ,
Pascale Vicat-Blanc Primet
oResearch Report N 2007-35
École Normale Supérieure de Lyon
46 Allée d’Italie, 69364 Lyon Cedex 07, France
Téléphone : +33(0)4.72.72.80.37
Télécopieur : +33(0)4.72.72.80.80
Adresse électronique : [e-mail]Towards a User-Oriented Benchmark for Transport Protocols
Comparison in very High Speed Networks
Romaric Guillier , Ludovic Hablot , Pascale Vicat-Blanc Primet
July 2007
Abstract
Standard TCP faces some performance limitations in very high speed wide
area networks, mainly due to a long end-to-end feedback loop and a conser-
vative behaviour with respect to congestion. Many TCP variants have been
proposed to overcome these limitations. However, TCP is a complex protocol
with many user-configurable parameters and a range of different implementa-
tions. It is then important to define measurement methods so that the transport
services and protocols can evolve guided by scientific principles and compared
quantitatively. The goal of this report is to present some steps towards a user-
oriented benchmark, called ITB, for high speed transport protocols compari-
son. We first present and analyse some results reported in the literature. From
this study we identify classes of ...

Sujets

• Run-time type information
• Protocole opératoire
• Itbayat
• Experiments and Observations on Different Kinds of Air
• Application linéaire
• Bottleneck

Informations

• Publié par: Undu
• Nombre de lectures: 37
• Langue: English
• Poids de l'ouvrage: 3 Mo

Extrait

Laboratoiredel'InformatiqueduParallélisme ÉcoleNormaleSupérieuredeLyon UnitéMixtedeRechercheCNRS-INRIA-ENSLYON-UCBLn o 5668 TowardsaUser-OrientedBenchmarkfor TransportProtocolsComparisoninveryHigh SpeedNetworks RomaricGuillier, LudovicHablot, PascaleVicat-BlancPrimet July2007 ResearchReportN o 2007-35 ÉcoleNormaleSupérieuredeLyon 46Alléed'Italie,69364LyonCedex07,France Téléphone:+33(0)4.72.72.80.37 Télécopieur:+33(0)4.72.72.80.80 Adresseélectronique: [e-mail]

TowardsaUser-OrientedBenchmarkforTransportProtocols ComparisoninveryHighSpeedNetworks RomaricGuillier,LudovicHablot,PascaleVicat-BlancPrimet July2007 Abstract StandardTCPfacessomeperformancelimitationsinveryhighspeedwide areanetworks,mainlyduetoalongend-to-endfeedbackloopandaconser- vativebehaviourwithrespecttocongestion.ManyTCPvariantshavebeen proposedtoovercometheselimitations.However,TCPisacomplexprotocol withmanyuser-congurableparametersandarangeofdifferentimplementa- tions.Itisthenimportanttodenemeasurementmethodssothatthetransport servicesandprotocolscanevolveguidedbyscienticprinciplesandcompared quantitatively.Thegoalofthisreportistopresentsomestepstowardsauser- orientedbenchmark,calledITB,forhighspeedtransportprotocolscompari- son.Werstpresentandanalysesomeresultsreportedintheliterature.From thisstudyweidentifyclassesofrepresentativeapplicationsandusefulmetrics. Wethenisolateinfrastructureparametersandtrafcfactorswhichinuecnethe protocolbehaviour.Thisenableustodenescenariocapturingandsynthesis- ingcomprehensiveandusefulproperties.Wenallyillustratethisproposalby preliminaryresultsobtainedonourexperimentalenvironment,Grid'5000,we havebuiltandareusingforcontributinginthisbenchmarkdesign. Keywords:ProtocolBenchmark,TCP,Performanceevaluation,HighSpeedtransport,HighSpeed networks Résumé

Laversion“standard”deTCPestconfrontéeàuncertainnombredelimita- tionsdeperformancedanslesréseauxàtréshautdébitquisontprincipale- mentcauséesparunebouclederétroactiondeboutenbouttroplongueetun comportementtrèsprudentvis-à-visdelacongestion.Ungrandnombrede variantesdeTCPontétéproposépourtenterdesurpasserceslimitations.Ce- pendantTCPestunprotocolecomplexecomportantbeaucoupdeparamêtres dénissablesparl'utilisateuretunéventaild'implémentationsdifférentes.Il estalorsimportantdedénirdesméthodesdemesureanquelesserviceset lesprotocolesdetransportpuissentévoluerselondesprincipesscientiques etêtrecomparésquantitativement.Lebutdecerapportestdeprésenterune démarcheversladénitiond'unbancd'essaiorientéutilisateur,appeléITB, pourlacomparaisondeprotocolesdetransportsdanslesréseauxàhautdébit. Nouscommençonsparprésenteretanalyserquelquesrésultatsprésentdans lalittérature.Apartirdecetteétude,nousidentionsdesclassesreprésenta- tivesd'applicationsetdesmétriquesutiles.Nousisolonsensuitelesparamêtres infrastructurelsetlesfacteursdetracquiontuneinuencesurlecomporte- mentdesprotocoles.Cecinouspermetdedénirdesscénariospermettantde captureretdesynthétiserdespropriétésutilesetcomplètes.Finalement,nous présentonsdesexemplesderésultatsobtenusdansl'environmentexpérimental Grid'5000illustrantnotredémarche. Mots-clés:Bancd'essaideprotocoles,TCP,évaluationdeperformances,transporthaut-débit, réseauxhaut-débit 2

BenchmarkforTransportProtocols1 Contents 1Introduction2 2Relatedwork3 2.1Transportprotocolsforhighspeednetworks......................3 2.2Evaluationframeworks.................................4 2.2.1EvaluationsonRealExperimentalNetworks.................5 2.2.2EvaluationsonEmulatedNetworks......................5 2.2.3EvaluationswithNS2simulator........................7 3DenitionandgoalsofaTransportBenchmarksuite7 3.1Whatisabenchmark?..................................7 3.2ExampleoftheNPBanditsusage...........................7 3.3Guidelinesfortransportprotocolbenchmarking....................8 4ITB:InriaTransportBenchmarkproposal8 4.1Representativeapplicationsselection..........................8 4.2Metrics.........................................9 4.2.1Metricstypes..................................9 4.2.2Throughput...................................9 4.2.3Fairness.....................................10 4.2.4Efciency....................................10 4.3Systemparameters...................................10 4.3.1Topology....................................11 4.3.2Capacities....................................11 4.4Workloadparameters..................................11 4.5Benchmark.......................................12 4.5.1WMapplicationcharacteristics.........................12 4.5.2PPapplicationcharacteristics.........................12 4.5.3BUapplicationcharacteristics.........................13 4.5.4PAapplicationcharacteristics.........................13 4.6ITBclassesdenition..................................13 5Realexperiments15 5.1Topologyexample....................................15 5.2TUapplication.....................................16 5.3BUapplication.....................................16 5.4PAapplication......................................18 5.5ITBparameters.....................................18 5.5.1Problemsize:RTTparameter.........................18 5.5.2Workloadparameters..............................21 5.6Metricsconsiderationexample.............................23 6Conclusion23 7Acknowledgement24

2R.Guillier,L.Hablot,P.Primet 1Introduction Today,mostdatatransferapplicationsandabout90%oftheInternettrafcusetheTCPprotocol. TCPhasshownagreatscalabilityinnumberofusers,butnotinlinkcapacity.TheTCPperformance canbeverylowandunstableindata-centerapplications,gridcomputingapplications,interactive communicationswithinhighspeedlongdistancenetworksinfrastructureslikebbertohome(FTTH) orlambdagridsenvironments.TheconservativebehaviourofTCPwithrespecttocongestioninIP networksisattheheartofthecurrentperformanceissuesfacedbythehigh-performancenetwork- ingcommunity.Severaltheoreticalandexperimentalanalysishaveshownthatthedynamicsofthe traditionalfeedbackbasedapproachistooslowinveryhighspeednetworksthatmaylosepackets. Consequentlynetworkresourceutilisationisnotoptimalandtheapplicationperformanceispoorand maybedisappointing.Manyhigh-endcomputingapplicationswishtotransferlargevolumesofdata overwideareanetworksandrequirehighdataratesinordertodoso.However,theseapplications arerarelyabletotakefulladvantageofthehigh-capacity(2.5Gbps,10Gbpsandupwards)networks installedtoday.DatafromInternet2showthat90%ofthebulkTCPows(denedastransfersofat least10MBofdata)uselessthan5Mbps,andthat99%uselessthan20Mbpsoutofthepossible 622Mbpsprovision.Therearemanyreasonsforsuchpoorperformance.Manyoftheproblemsare directlyrelatedtotheendsystem,totheprocessorandbusspeed,andtotheNICwithitsassociated driver.TCPconguration(e.g.smallbufferspace)hasalsoasignicantimpact.Butwhenthese problemsarexed,thecongestioncontrolalgorithmisoneofthekeycomponentwhichhastobe modiedtoalleviatetheperformanceprobleminhighspeedlongdistancenetworksenvironments. Congestioncontrolisthemostimportantandcomplexpartofatransportprotocolinpacketswitch sharednetwork.TCPprovidesafullydistributedcongestioncontrolprotocolwhichstatisticallyshare availablebandwidthamongowsfairly.TCPwasdesignedrstandforemsottoberobustandwhen congestionisdetected,TCPsolvestheproblembutattheexpenseofperformance.Forexample,for astandardTCPconnectionwith1500-bytepacketsanda100msround-triptime,achievingasteady- statethroughputof10Gbpswouldrequireanaveragecongestionwindowof83,333segments,anda packetdroprateofatmostonecongestioneventevery5,000,000,000packets(orequivalently,atmost onecongestioneventevery12/3hours). Tosolvethisproblemseveralprotocolsenhancementshavebeenproposed[WHVBPa05,SL04, XHR04].Alltheseprotocolsarenotequivalentandnotsuitedforallenvironments.Someofthe protocolstargetinghighspeedInternet,attempttoimproveTCPresponsefunctionwhiletryingto retainmaximumbackwardscompatibilitywithlegacyimplementations.Othersfocusondifferent targetenvironments,forexamplededicatedopticalnetworks.Theyarelessconservativeandtheycan beimplementedinuserspaceandoverUDP. Sinceacoupleofyears,theevaluationandcomparisonoftheseprotocolsreceiveanincreasing amountofinterest.However,TCPandotheralternativesarecomplexprotocolswithmanyuser- congurableparametersandarangeofdifferentimplementations.Severalaspectscanbestudiedand varioustestingmethodsexist.Theresearchcommunity 1 recogniseitisimportanttodeploymeasure- mentmethodssothatthetransportservicesandprotocolscanevolveguidedbyscienticprinciples. Researchersanddevelopersneedagreed-uponmetrics-acommonlanguageforcommunicatingre- sults,sothatalternativeimplementationscanbecomparedquantitatively.Usersofthesevariantsneed performanceparametersthatdescribeprotocolcapabilitiessothattheycandevelopandtunetheir applications.Protocoldesignersneedexamplesofhowuserswillexercisetheirservicetoimprove thedesign.Thegoalofthisreportisthentocontributetothiseffortandpresentsomestepstowardsa 1 Seattleworkshop,February2007

BenchmarkforTransportProtocols3 user-orientedbenchmarkdesignforhighspeedtransportprotocolscomparison. Therestofthereportisorganisedasfollows.Relatedworksandongoingeffortsarereviewed inSection2.Section3denesthenotionofbenchmarkandintroducesafewexamplesofsuch tools.Section4introducesthemetrics,parametersandmeasurementmethodsconstitutingourbench- markpropositionfortransportprotocols.Section5illustratesthisproposalwithpreliminaryresults obtainedonourexperimentalenvironment,Grid'5000,wehavebuiltandareusingforcontributingin thisbenchmarkdesign.Finally,weconcludeinSection6andproposesomeperspectivesforprotocol andnetworkserviceenhancement. 2Relatedwork HighSpeedtransportprotocoldesignandevaluationisahotresearchtopic[VBTK06].Werst overviewandclassifyproposedalternativestoTCPandthensurveyseveraleffortstowardsasystem- aticevaluationoftheseproposals. 2.1Transportprotocolsforhighspeednetworks TherecentalternativestoTCP,dedicatedtohighspeedInternet,aimatsolvingtheproblemofthepoor responsefunctionofTCPinlargebandwidth-delayproductnetworksbymodifyingtheparameters, α fortheincreaseand β forthedecrease,oftheAdditiveIncreaseMultiplicativeDecrease(AIMD) algorithmthatisusedduringthecongestionavoidancephaseofTCP.ForexampleHighSpeedTCP [Flo03]andScalableTCP[Kel03]increasetheaggressivenessinhigh-performancecontextswhile tryingtostayfairtostandardTCPowsinlegacycontexts.Table2summarisestheAIMDvalues thatareusedbydifferentTCPvariants.In[PFTK98],Padhyeetal.presentasimpleTCPmodelto expressthethroughputasafunctionoftheRTT,segmentsize,AIMDconstantsandlossprobability: q MSS q (1) RTT ∗ 2 3 p + RTO ∗3 3 8 p p (1+32 p 2 ) 3 SSM thatcanbereducedto r R = RTT 2 p (2) ifweassumethatthelossrateissmall(typicallythecaseintheopticalnetworksthatformsthe backboneofmostgrids/datacenters).ThesamekindofapproachhasbeenmadeforthenewTCP variants[Xu07]andsimilarexpressionsexist.Table1providesanapproximatevalueofthe c and d parametersconsideringthatalltheformulaareintheform cSSM R = RTTp d (3) . H-TCPissupposed[SL04]tohavethesameresponsefunctionasHighSpeedTCP.CUBICisnot includedintheTable1asitsresponsefunctiondoesn'ttintheformula.BychoiceoftheAIMD constants,ithasthesameresponsefunctionasTCPRenoforasmallcongestionepoch/BDPvalueto ensureitsfairnesswithrespecttothisprotocol.Theseresponsefunctionsprovideaninsightofthe relativeperformanceofeachTCPvariantsforagivenBDPvalue,butthismodeldoesn'tcapturea numberofcharacteristicsofrealnetworkslikereversetrafcormultiplexing.ManyoftheseTCP

4 TCPvariantcd TCPReno1.220.5 BIC15.50.5 HighSpeedTCP0.120.835 Scalable0.081.0 R.Guillier,L.Hablot,P.Primet Table1:TCPvariants'responsefunctionparameters TCPvariant αβ TCPReno 1 2 1 BIC 1 or bin.search 8 1 CUBIC cub ( cwnd,history ) 5 1 HighSpeedTCP inc ( cwnd ) decr ( cwnd ) HamiltonTCP f ( last l oss )1− R R T T T T m m a in x ScalableTCP 0 . 01∗ cwnd 8 1 Table2:TCPvariants'AIMDconstants variantsareavailableinallrecentLinuxkernel:HighSpeedTCP[Flo03],ScalableTCP[Kel03], Hamilton-TCP[SL04],BIC[XHR04],CUBIC[RX05]andtheycanbeusedbyeveryone. TCPVegas[BOP94]andFAST-TCP[DXWH07]useothercongestioninformationavailable (round-triptimevariations,ExplicitCongestionNotication,etc)toregulatethroughputatthesender endandthusnelycontrolbufferllinginrouters,managingIPcongestinooptimally.XCPgoes furtherfromtoday'sstandards,proposinganewcooperativecongestioncontrolschemefeaturinga precisecongestionwindowindicationgoingfromrouterstoendhosts. UDT,aUDP-basedDataTransferprotocol[GG07]addresstheproblemoftransferringlargevol- umetricdatasetsoverhighbandwidth-delayproductopticalnetworks.LikesomeTCPvariantssuch as[XHR04],UDTemploysanewwindow-basedcongestioncontrolalgorithmtargetingatuncon- trolledsharednetworks. 2.2Evaluationframeworks Sinceacoupleofyearsseveralteamsaimatdevelopingmethodologiesandtoolsprovidingcompre- hensivestandards-compliancetestingofTCPimplementations.Inthissection,wepresentinitiatives focusingonTCPvariantevaluation.Varioustestingmethodsexisttoevaluatetransportprotocolper- formance:realInternet,realexperimentalnetworks,emulatednetworks,simulation.Eachonehas itspitfalls.Amixofseveralmethodsishighlyrequiredtoproduceconvincingresults[All99].To ourknowledge,therealInternet,throughthePlanetlabtestbedforexample,hasnotbeenemployed toevaluateextensivelythenewTCPvariants.Severalmethodologiesandresultshavebeenproposed by[LLS06,Flo06,HLRX06]toidentifycharacteristicsanddescribewhichaspectofevaluationsce- nariodeterminethesecharacteristicsandhowtheycanaffecttheresultsoftheexperiments. Thenextsectionscomparerelatedworksaccordingtothetypeofmethodtheyhaveadopted.

BenchmarkforTransportProtocols5 2.2.1EvaluationsonRealExperimentalNetworks Fewrealexperimentshavebeenran[CAK+05,LLS06]toanalysethebehaviourofarangeofnew protocolsinhighspeedInternetcontext.Otherrecentworkfocusonsharedhighspeednetworks dedicatedtohighperformancedistributedapplications[GHK+07,GSP07,GHK+06].Therealex- perimentmethodgivesarealinsightoftheprotocolbehaviourinveryhighspeedenvironment(e.g. 10Gbps),explorestheinteractionswiththehardwareinfrastructureandgenerallyhelpstodebugthe globalhardwareandsoftwarecommunicationchain. Wan-In-LabWan-In-Lab[GSLL07]isatestbedoftheCaliforniaInstituteoftechnology.Itisbuilt arounda2400kmopticbbercableandarraysofopticalswitchestoconstructnetworkswithvariable lengthandRTT.Itisaccessibletousersthroughawebinterface.Thisinterfaceallowsuserstoupload experimentalkernels,instrumentedwiththeWeb100tools,andtorunasetofpredenedtests.The resultsofthesetestsarethenprocessedandplacedonthewebinbothgraphicalandnumericalform. ProtocolsaretestedforRTTfairness,convergencespeedbothwithandwithoutexistinglargeows, interactionwithshortows,andfairnessbetweenowstraversingdiffreentnumbersofhops. Theyareconsideringarangeofexperimentscombiningatopologyandsomenetworkingcondi- tionstostudyinterestingcases 2 .Themaininterestofthistestbedisthepossibilitytoperformreal experimentsusingreallinks(1to10Gbpsspeed)andopticalswitchesandtohaveaccesstoahuge rangeofRTTs(from0to180mswith2msincrements)byconguringtheopticalswitches. Grid5000Grid'5000project[BCC+06],isanexperimentalgridplatformgathering2500proces- sorsoverninegeographicallydistributedsitesinFrance.Thenetworkinfrastructureisanintercon- nectionofLANs(i.e.gridsites)andan10Gbpsopticalvirtualprivatenetwork(VPN).Asimplied topologyisshowninFigure1.Theparticularityofthistestbedistoprovideresearcherswithafully recongurabilityfeaturetodynamicallydeployanyOSimageorTCPstackonanyendhostofthe testbedandwithafullydedicatedopticalnetwork. ThistestbedhasbeenusedforexperimentingdifferentTCPstacksandseveraltypesofworkload correspondingtorealisticgridcomputinganddata-centerapplications[GHK+07,GSP07].Internet- liketrafccanalsobeinjectedinthistestbed. 2.2.2EvaluationsonEmulatedNetworks Deploymentofrealnetworksiscostlyandexperimentscanbetimeconsuming.Moreover,such testbedshardlyproviderangeoflatenciestofullyexploreprotocolbehaviour.Forthesereasons, severalteamsadoptemulationmethodbyusingsoftwareorhardwarenetworkemulators. HamiltonUniversityframeworkWiththeirlatestexperimentsonTCP[LLS06],DoughLeithet al.presentanexperimentaltest-bed,basedaroundtheDummynet[Riz97]networkemulator.The goalistostudytheperformanceofvariousTCPvariantsandtoproposeofaseriesofbenchmarktests easilyreproducible.ThetopologyusedisaclassicaldumbbellwithaDummynetrouterinthemiddle toemulatelatencyandtosetqueuesizeandbottleneckspeed Asetofscripts 3 ,alongwiththeresultsofalargequantityofexperimentswithgraphsaremade available.Theyareconsideringawiderangeofparametersincluding:queuesize,bottlenecksize, 2 http://wil.cs.caltech.edu/mwiki/index.php?title=Experiments 3 workingwithTCP-Linux[WC06],aNS-2patchtouseGNU/LinuxTCPcongestioncontrolalgorithmmodulesto performexperiments

6 10 GbE Links   1 GbE Links Lille Orsay Nancy Rennes R.Guillier,L.Hablot,P.Primet noyL Grenoble Bordeaux Toulouse Sophia Figure1:Grid'5000backbone RTT(16,22,42,82,162ms),asymmetricRTT,TCPvariantsandnumberofwebsessions.Theavailable testsweredonewithtwoowsandamaxbottlenecksizeof250Mbps. NorthCarolinaStateUniversityframeworkFortheexperiments[HLRX06]andvalidationof theirTCPvariants 4 ,InjongRheeetal.useatestbedalsobasedonDummynet.TwoDummynetrouters areused,onetomanagetheAQMandthebandwidthlimitation,thesecondtoadddelay,forminga dumbbelltopology.Thechoiceofthistopologyismotivatedbythefactthatevenamorecomplicated topologylikeaparkinglotfailstocapturetherealismofproductionnetworks.Thebottleneckisxed to400MbpsandtheRTTvaluesusedare16,64,162and324ms. Twokindsofbackgroundtrafcmaybeinjected:short-livedandlonglived.Theshort-lived trafcisgeneratedwithacustomversionoftheSURGE[BC98]webtrafcgeneratorfollowing5 differentkindoftrafcdistribution.Thelong-livedtrafcisgeneratedusingiperf.Theamountof backgroundtrafcissettobeabout70Mbps(lessthan20%ofthebottleneckcapacity).Thebasic scenarioconsistsinmonitoringtheinteractionsoftwoowswhentheyareinteractingwithvarious levelsofbackgroundtrafc. AIST-INRIAframeworkTheAISTandINRIAteamsusehardwareemulatorscombinedwithnet- workvirtualisationsoftwareeWANtoevaluateprotocolsunderdifferentlatencyandtopologycon- ditions[PTK+06].AIST-GtrcNET-10isahardwareemulatorthatallowslatencyemulationupto 858mswithoutlosses,ratelimitationandprecisebandwidthmeasurementsat10Gbpswirespeed. GtrcNET-10p3isafullyprogrammablenetworktestbed,whichisa10Gbpssuccessorofawell- establishednetworktestbed,GtrcNET-1. 4 BICandCUBICTCP

BenchmarkforTransportProtocols7 2.2.3EvaluationswithNS2simulator TMRGNS-2frameworkIntheIRTFdraft[NS207],WeiandFloydproposeaframeworkof benchmarkingTCPvariantsbasedontheNS2networkemulator.Itdenestopologies,trafccharges andmetricsthatcouldbeusedtoevaluatetheperformanceofTCPstacks. Currently,threetopologiesareconsidered:dumbbell(onebottleneck),parking-lot(multiplebot- tleneck)withcrosstrafcanda“simplenetwork”topologywithtransitandsbtudomains.Fourkind oftrafcmodelsareproposed:FTPtrafc(long-livedows),Webtrfac(short-livedows),video streamingtrafc(CBRtrafcoverUDP)andvoicetrafc(CBRorON/OFFows).Metricsdened intheTRMG'smetricsdraft[Flo07]areconsidered:throughput,queueingdelay,jitter,lossrate, responsetime,fairness,convergenceandrobustness.Eachkindofmetricsisadaptedtothetraf- cmodelitiscurrentlytryingtomeasure.Astheyaredescribingaframework,theydon'tprovide explicitscenariithatmightbeinterestingtorun. PolitecnicodiBaristudiesMascoloin[MV06]isusingNS-2simulationstoobservetheimpact ofreversetrafconthenewTCPcongestioncontrolalgorithms.Thetestbedisbasedonadumbbell topologywithatmost6differentsources/destinations.Thebottlenecksizeis250Mbpsandisalso sharedbytwonetworkstransmittingwebtrafc.Heismainlyfocusingontheproleoftheworkloads thatareappliedtothesystem(on/offreversetrafc,reversetrafc+webtrafc,reversetrafc+web trafc+differentRTTs).HeisusingthefollowingRTTsvalues:40,80and160ms.Themetrics usedtoanalysetheresultsaremainlylinkutilisation,goodput,congestionwindowsizeandtimeouts events. Thisshortoverviewshowsthatsoftwaretestbeds(simulatorsoremulators)enablecomplextopolo- gies,largenumberofowsexperimentsbutthebottleneckcapacityislimitedto400Mbpsandthe latenciesto400ms.Hardware-basedtestbedsgiveresearchersaccessto10Gbpsandupto800ms latenciesbutpresentsomelimitationintopologiescomplexity. 3DenitionandgoalsofaTransportBenchmarksuite 3.1Whatisabenchmark? Abenchmarkisaprogramorasetofprograms,whichcalculatetherelativeperformanceofamachine oranarchitecture(hardware),oranotherprogram(software).Eachbenchmarkmayeitherfocuson quantity(executionspeed,amountofdatacomputed,etc.)oronquality(robustness,security,etc.). Benchmarkcanbeexecutedattwodifferentlevels: • lowlevelormicrobenchmarks:Testingtheperformanceofoneparticularcomponentorfunc- .noit • applicationlevel:Aimingatrepresentingtypicalapplications/workloadsofaplatformthat needsevaluation.AmongtheexistingHighPerformanceComputing(HPC)benchmarks,the NASParallelBenchmark(NPB)[FY02]iswellknown.Thissetofprogramsrepresentsthe typicalapplicationsclassesexecutedonclusters. 3.2ExampleoftheNPBanditsusage ( In BT H , P C C G , , th E e P, N F P T B ,I [ S F , Y L 0 U 2 , ]i M s G a a c n o d m S m P o ) n t l h y at us g e i d ve b s e a nc g h o m od ar p k. an T el he of N th P e B d i i s ff a er g e r n o t u p p ar o a f ll e e i l g a ht pp p l r i o ca g t r i a o m ns s