Benchmarks
13 pages
English

Benchmarks

Le téléchargement nécessite un accès à la bibliothèque YouScribe
Tout savoir sur nos offres
13 pages
English
Le téléchargement nécessite un accès à la bibliothèque YouScribe
Tout savoir sur nos offres

Description

1 Benchmarks
The following requirements apply to all benchmarks. If a benchmark program spans
multiple machines, each machine must satisfy all of these requirements.
• Times are measured in real time, with a resolution of no worse than one second.
• The benchmark program and all related processes may use no more than 75%
of the machine’s physical memory, and in no event more than two gigabytes of
physical memory.
• The benchmark program and all related processes may use no more than ten
gigabytes of disk space.
• The benchmark program must produce the correct answer.
• The program must not encounter an error or prematurely terminate
during or after the benchmark.
• The benchmark program must be in a usable condition after the benchmark.
• The program may not degrade or disable the user interface, or in other
manner cause less work to be done than as expected by normal user interaction.
1.1 Test Environment
Processor Intel Pentium 4 stepping D1
3.2 GHz clock speed
800 MHz front side bus speed
Cache 8 KB L1 data
12 KB L1 instruction
512 KB L2 unified
Memory 1 GB DDR 2700 RAM
Disk 80 GB capacity
100 MB/s interface bandwidth
49.3 MB/s average transfer rate
19.2 ms access time
Operating System Microsoft Windows XP Service Pack 1a
Table 1: Machine Configuration
The processor HyperThreading extension was enabled during the benchmarks al
though this does not significantly affect performance. Almost all of the benchmarking
work is done on a single thread.
Execution time was ...

Sujets

Informations

Publié par
Nombre de lectures 50
Langue English

Exrait

1BenchmarksThefollowingrequirementsapplytoallbenchmarks.Ifabenchmarkprogramspansmultiplemachines,eachmachinemustsatisfyalloftheserequirements.Timesaremeasuredinrealtime,witharesolutionofnoworsethanonesecond.Thebenchmarkprogramandallrelatedprocessesmayusenomorethan75%ofthemachine’sphysicalmemory,andinnoeventmorethantwogigabytesofphysicalmemory.Thebenchmarkprogramandallrelatedprocessesmayusenomorethantengigabytesofdiskspace.Thebenchmarkprogrammustproducethecorrectanswer.Thebenchmarkprogrammustnotencounteranerrororprematurelyterminateduringorafterthebenchmark.Thebenchmarkprogrammustbeinausableconditionafterthebenchmark.Thebenchmarkprogrammaynotdegradeordisabletheuserinterface,orinothermannercauselessworktobedonethanasexpectedbynormaluserinteraction.1.1TestEnvironmentProcessorIntelPentium4steppingD13.2GHzclockspeed800MHzfrontsidebusspeedCache8KBL1data12KBL1instruction512KBL2uniedMemory1GBDDR2700RAMDisk80GBcapacity100MB/sinterfacebandwidth49.3MB/saveragetransferrate19.2msaccesstimeOperatingSystemMicrosoftWindowsXPServicePack1aTable1:MachineCongurationTheprocessorHyperThreadingextensionwasenabledduringthebenchmarksal-thoughthisdoesnotsignicantlyaffectperformance.Almostallofthebenchmarkingworkisdoneonasinglethread.Executiontimewasdeterminedbythesystemhighperformancetimer.Thehighperformancetimerhasaresolutionofoneprocessorclockcycle.Theruntimeenvi-ronmentwasconguredtoterminatethebenchmarkprogramifmorethan75%(768MB)ofphysicalmemorywasallocatedfromtheheap.Duringbenchmarkexecution,thepeakheapallocationsizewasrecorded.Notethatthisvaluemaybemorethan1
ProgramJigCellversion04302004CompilerIBMJikesJavacompilerversion1.19OptimizerSunJavaHotSpotClientVMversion1.4.2.03-b02,mixedmodeRuntimeSunJavaRuntimeEnvironmentSEversion1.4.2.03-b02Table2:SoftwareCongurationtheamountofmemoryrequiredbythebenchmarkasfreedobjectsneednotbeimme-diatelyreclaimedbytheruntimeenvironmentifmemoryisavilable.Additionally,alowerboundonthememoryrequiredwasobtainedbydecreasingtheamountofheapmemoryavailabletotheruntimeenvironmentuntilthebenchmarkwasunabletocom-pleteduetomemoryexhaustion.1.2ModelTestDataAmodelconsistsofthefollowingelements:ModelnameListofcompartments.Eachcompartmenthasanameanddescription.Listofreactions.Eachreactionhasaname,description,kineticinformation,andparameters.ThekineticequationforrectioniisdSdit+1=kiSiki+1Si+1startingwithi=0.Eachparameterhasanameanddescription.Thevalueofparameteriisi+12.Listofspeciesusedinreactions.Eachspecieshasanameanddescription.Thesizeofamodelisdeterminedbythenumberofreactions.1.3ModelLoadingPerformanceModelloadingperformancemeasuresthesizeofmodelssupportedbytheModelBuilder.Amodelissupportediftheloadingtimeislessthan300seconds.TheModelBuildermustsupportamodelcontaining1000reactionstosatisfythemodelbuildingperfor-mancerequirements.Theprocedureformeasuringmodelloadingperformanceis:1.CreateatestmodelcontainingNreactions2.Writetestmodeltodisk3.ReadtestmodelfromdiskStep3oftheprocedureistimed.AsofMay1st,theJigCellModelBuilderdoesnotpassthisbenchmark.Thetimetoloadamodelcontaining1000reactionsexceeds626secondsusinglessthan720MBofmemoryand686KBofdiskspace.2
1.4ReactionEntryPerformanceReactionentryperformancemeasuresthetimerequiredtoaddanewreactiontothemodel.TheModelBuildermustsupportaddingareactiontoamodelcontaining200reactionsin120secondstosatisfythemodelbuildingperformancerequirements.Theprocedureformeasuringreactionentryperformanceis:1.CreateatestmodelcontainingNreactions2.Writetestmodeltodisk3.Readtestmodelfromdisk4.AddspeciesN+15.AddreactionN+16.AddparameterN+17.WritetestmodeltodiskSteps3through7oftheprocedurearetimed.AsofMay1st,theJigCellModelBuilderhasnotbeenrunforthisbenchmark.1.5RunTestData1.6RunSetLoadingPerformanceRunsetloadingperformancemeasuresthesizeofrunsets,parametersets,andinitialconditionsetssupportedbytheModelBuilder.Asetissupportediftheloadingtimeislessthan300seconds.TheModelExecutormustsupportarunsetcontaining10000runs,500parametersets,and500initialconditionsetstosatifsythemodelexecutionperformancerequirements.TheModelExecutordoesnotneedtosupporttheselimitssimultaneously.Therearethreeproceduresformeasuringrunsetloadingperformance.Therstproceduremeasuresrunsetloadingperformancewiththemaximumrunsetsize:1.Createatestmodelcontaining200reactions2.Writetestmodeltodisk3.Create1testparameterset4.Create1testinitialconditionset5.CreateatestrunsetcontainingNruns6.Writetestrunsettodisk7.Readtestrunsetfromdisk3
Thesecondproceduremeasuresrunsetloadingperformancewiththemaximumparametersetsize:1.Createatestmodelcontaining200reactions2.Writetestmodeltodisk3.CreateNtestparametersets4.Create1testinitialconditionset5.CreateatestrunsetcontainingNruns6.Writetestrunsettodisk7.ReadtestrunsetfromdiskThethirdproceduremeasuresrunsetloadingperformancewiththemaximumini-tialconditionsetsize:1.Createatestmodelcontaining200reactions2.Writetestmodeltodisk3.Create1testparameterset4.CreateNtestinitialconditionsets5.CreateatestrunsetcontainingNruns6.Writetestrunsettodisk7.ReadtestrunsetfromdiskStep7ofeachoftheseproceduresistimed.AsofMay1st,theJigCellRunManagerhasnotbeenrunforthesebenchmarks.1.7RunEntryPerformanceRunentryperformancemeasuresthetimerequiredtoaddanewruntoarunset.TheModelExecutormustsupportaddingaruntoarunsetcontaining2000runsin300secondstosatisfythemodelexecutionperformancerequirements.1.Createatestmodelcontaining200reactions2.Writetestmodeltodisk3.Create100testparametersets4.Create100testinitialconditionsets5.CreateatestrunsetcontainingNruns4
6.Writetestrunsettodisk7.Readtestrunsetfromdisk8.AddrunN+19.WritetestrunsettodiskSteps7through9oftheprocedurearetimed.AsofMay1st,theJigCellRunManagerhasnotbeenrunforthisbenchmark.1.8RunExecutionPerformanceRunexecutionperformancemeasuresthetimerequiredtoexecuteanexistingrun.TheModelExecutormustsupportexecutingarunforaworkingmodelfromarunsetcontaining2000runsin60secondstosatisfythemodelexecutionperformancerequirements.Theprocedureformeasuringrunexecutionperformanceis:1.Createatestmodelcontaining200reactions2.Writetestmodeltodisk3.Create100testparametersets4.Create100testinitialconditionsets5.CreateatestrunsetcontainingNruns6.Writetestrunsettodisk7.Readtestrunsetfromdisk8.ExecuterunNSteps7and8oftheprocedurearetimed.AsofMay1st,theJigCellRunManagerhasnotbeenrunforthisbenchmark.1.9RunModicationPerformanceRunmodicationperformancemeasuresthetimerequiredtomodifyanexistingrun.TheModelExecutormustsupportmodifyingarunforaworkingmodelfromarunsetcontaining2000runsin60secondstosatisfythemodelexecutionperformancerequirements.Theprocedureformeasuringreactionmodicationperformanceis:1.Createatestmodelcontaining200reactions2.Writetestmodeltodisk3.Create100testparametersets5
4.Create100testinitialconditionsets5.CreateatestrunsetcontainingNruns6.Writetestrunsettodisk7.Readtestrunsetfromdisk8.Settheparameterforreaction200inparameterset100to09.Settheinitialconditionforspecies200ininitialconditionset100to010.WritetestrunsettodiskSteps7through10oftheprocedurearetimed.AsofMay1st,theJigCellRunManagerhasnotbeenrunforthisbenchmark.1.10ExperimentSetTestDataAnexperimentsetconsistsofexperiments.Eachexperimenthasthefollowingele-ments:ExperimentnameGloballyuniqueidentierExperimentalobservations.Theexperimentalvalueisatwovariabletimeseriescontaining4observations.Therstvariableofexperimentobservationiistheintegeriandthesecondvariableisanite,normalizedrealnumber.Experimentaltype.Thetypemustbecompatiblewiththetimeseriesformatoftheexperimentalvalue.ExperimentdescriptionThesizeofanexperimentsetisdeterminedbythenumberofexperiments.1.11ExperimentLoadingPerformanceExperimentloadingperformancemeasuresthesizeofexperimentsetssupportedbytheModelAnalyzer.Anexperimentsetissupportediftheloadingtimeislessthan300seconds.TheModelAnalyzermustsupportanexperimentsetcontaining10000experimentstosatisfythemodelanalysisperformancerequirements.Theprocedureformeasuringexperimentloadingperformanceis:1.CreateatestexperimentsetcontainingNexperiments2.Writetestexperimentsettodisk3.ReadtestexperimentsetfromdiskStep3oftheprocedureistimed.AsofMay1st,theJigCellComparatorpassesthisbenchmark.Thetimetoloadanexperimentsetcontaining10000experimentsislessthan5secondsusinglessthan64MBofmemoryand5011KBofdiskspace.6
1.12ExperimentEntryPerformanceExperimententryperformancemeasuresthetimerequiredtoaddanewexperimenttoanexperimentset.TheModelAnalyzermustsupportaddinganexperimenttoanex-perimentsetcontaining2000experimentsin120secondstosatisfythemodelanalysisperformancerequirements.Theprocedureformeasuringexperimententryperformanceis:1.CreateatestexperimentsetcontainingNexperiments2.Writetestexperimentsettodisk3.Readtestexperimentsetfromdisk4.AddexperimentN+15.WritetestexperimentsettodiskStep3through5oftheprocedureistimed.AsofMay1st,theJigCellComparatorpassesthisbenchmark.Thetimetoaddanewexperimenttoanexperimentsetcontaining2000experimentsislessthan3secondsusinglessthan64MBofmemoryand998KBofdiskspace.1.13TransformSetTestDataAtransformsetconsistsoftransformsandamappingfunctionfromexperimentnamestotransforms.Eachtransformhasthefollowingelements:TransformnameGloballyuniqueidentierTransformationprocedure.Thetransformationprocedureisafunctionalspeci-cationofwhatthetransformdoes.ParametersusedbythetransformationprocedureReferencetothetransformthatprovidesinputtothistransform,ifneededListoftransformscontainedinthistransformTheproceduredenedforeachnamedexperimentconsistsoftwotransformsandrequiresoneparameter.Thersttransformreceivedinputfromthesecondtransform;thesecondtransformdoesnotrequireinput.Neithertransformcontainsanytrans-forms.Theoutputoftheprocedureisatimeseriesofthesameformatasanexperimen-talobservation.Thesizeofatransformsetisdeterminedbythenumberofmappingsfromexperimentnamestotransforms.7
1.14TransformLoadingPerformanceTransformloadingperformancemeasuresthesizeoftransformsetssupportedbytheModelAnalyzer.Atransformsetissupportediftheloadingtimeislessthan300sec-onds.TheModelAnalyzermustsupportatransformsetfor10000experiments,ofwhichatleast500ofthetransformsaredifferent,tosatisfythemodelanalysisperfor-mancerequirements.Whenrunningthisbenchmark,allofthegeneratedtransformsaredifferent.Theprocedureformeasuringloadingloadingperformanceis:1.CreateatestexperimentsetcontainingNexperiments2.Writetestexperimentsettodisk3.CreateatesttransformsetcontainingNtransforms4.Writetesttransformsettodisk5.Readtestexperimentsetfromdisk6.ReadtesttransformsetfromdiskStep6oftheprocedureistimed.AsofMay1st,theJigCellComparatorpassesthisbenchmark.Thetimetoloadatransformsetfor10000experimentsislessthan25secondsusinglessthan256MBofmemoryand15971KBofdiskspace.1.15ObjectiveSetTestDataAnobjectivesetconsistsofobjectivesandamappingfunctionfromexperimentnamestoobjectives.Eachobjectivehasthefollowingelements:ObjectivenameGloballyuniqueidentierObjectiveprocedure.Theobjectiveprocedureisafunctionalspecicationofwhattheobjectivedoes.ParametersusedbytheobjectiveprocedureTheproceduredenedforeachnamedexperimentrequiresoneparameter.Thesizeofanobjectivesetisdeterminedbythenumberofmappingsfromexperimentnamestoobjectives.8
1.16ObjectiveFunctionLoadingPerformanceObjectivefuntionloadingperformancemeasuresthesizeofobjectivesetssupportedbytheModelAnalyzer.Anobjectivesetissupportediftheloadingtimeislessthan300seconds.TheModelAnalyzermustsupportanobjectivesetfor10000experiments,ofwhichatleast50oftheobjectivefunctionsaredifferent,tosatisfythemodelanal-ysisperformancerequirements.Whenrunningthisbenchmark,allofthegeneratedobjectivefunctionsaredifferent.Theprocedureformeasuringobjectivefunctionloadingperformanceis:1.CreateatestexperimentsetcontainingNexperiments2.Writetestexperimentsettodisk3.CreateatesttransformsetcontainingNtransforms4.Writetesttransformsettodisk5.CreateatestobjectivesetcontainingNobjectives6.Writetestobjectivesettodisk7.Readtestexperimentsetfromdisk8.Readtesttransformsetfromdisk9.ReadtestobjectivesetfromdiskStep9oftheprocedureistimed.AsofMay1st,theJigCellComparatorpassesthisbenchmark.Thetimetoloadanobjectivesetfor10000experimentsislessthan10secondsusinglessthan256MBofmemoryand15169KBofdiskspace.1.17ExperimentCongurationPerformanceExperimentcongurationperformancemeasuresthetimerequiredtocongurethetransformandobjectivefunctionforanewexperiment.TheModelAnalyzermustsupportconguringanexperimentinanexperimentsetcontaining2000experimentsin300secondstosatisfythemodelanalysisperformancerequirements.Theprocedureformeasuringexperimentcongurationperformanceis:1.CreateatestexperimentsetcontainingN+1experiments2.Writetestexperimentsettodisk3.CreateatesttransformsetcontainingNtransforms4.Writetesttransformsettodisk5.CreateatestobjectivesetcontainingNobjectives9
6.Writetestobjectivesettodisk7.Readtestexperimentsetfromdisk8.Readtesttransformsetfromdisk9.CreatetransformN+110.AddtransformN+1toexperimentN+111.Writetesttransformsettodisk12.Readtestobjectivesetfromdisk13.CreateobjectiveN+114.AddobjectiveN+1toexperimentN+115.WritetestobjectivesettodiskSteps7through16oftheprocedurearetimed.AsofMay1st,theJigCellComparatorpassesthisbenchmark.Thetimetocon-gureanexperimentinanexperimentsetcontaining2000experimentsislessthan20secondsusinglessthan256MBofmemoryand5207KBofdiskspace.1.18SummaryofResultsAsofMay1st,6ofthe13benchmarkshavebeencompletedandrun.JigCellpasses5ofthebenchmarksandfails1ofthebenchmarks.#Time(s)Memory(MB)Disk(KB)Passes1.3626.28512-7206864.1a6.1b6.1c6.17.18.19.11.115.0032-4950111.122.8932-419981.1424.03128-187159711.169.94128-150151691.1719.4532-2565207Table3:BenchmarkResultsSummary01
FigureFigure:1:2BenchmarkBenchmark113.111.1
  • Accueil Accueil
  • Univers Univers
  • Ebooks Ebooks
  • Livres audio Livres audio
  • Presse Presse
  • BD BD
  • Documents Documents