Benchmarks

Description

1 Benchmarks
The following requirements apply to all benchmarks. If a benchmark program spans
multiple machines, each machine must satisfy all of these requirements.
• Times are measured in real time, with a resolution of no worse than one second.
• The benchmark program and all related processes may use no more than 75%
of the machine’s physical memory, and in no event more than two gigabytes of
physical memory.
• The benchmark program and all related processes may use no more than ten
gigabytes of disk space.
• The benchmark program must produce the correct answer.
• The program must not encounter an error or prematurely terminate
during or after the benchmark.
• The benchmark program must be in a usable condition after the benchmark.
• The program may not degrade or disable the user interface, or in other
manner cause less work to be done than as expected by normal user interaction.
1.1 Test Environment
Processor Intel Pentium 4 stepping D1
3.2 GHz clock speed
800 MHz front side bus speed
Cache 8 KB L1 data
12 KB L1 instruction
512 KB L2 unified
Memory 1 GB DDR 2700 RAM
Disk 80 GB capacity
100 MB/s interface bandwidth
49.3 MB/s average transfer rate
19.2 ms access time
Operating System Microsoft Windows XP Service Pack 1a
Table 1: Machine Configuration
The processor HyperThreading extension was enabled during the benchmarks al
though this does not significantly affect performance. Almost all of the benchmarking
work is done on a single thread.
Execution time was ...

Sujets

• Linternaute.com
• Réaction de Wurtz
• Objectivité
• Le Modèle millionaire
• Modelica
• Carte MS

Informations

• Publié par: Urdie
• Nombre de lectures: 50
• Langue: English

Extrait

1Benchmarks Thefollowingrequirementsapplytoallbenchmarks.Ifabenchmarkprogramspans multiplemachines,eachmachinemustsatisfyalloftheserequirements. • Timesaremeasuredinrealtime,witharesolutionofnoworsethanonesecond. • Thebenchmarkprogramandallrelatedprocessesmayusenomorethan 75 % ofthemachine’sphysicalmemory,andinnoeventmorethantwogigabytesof physicalmemory. • Thebenchmarkprogramandallrelatedprocessesmayusenomorethanten gigabytesofdiskspace. • Thebenchmarkprogrammustproducethecorrectanswer. • Thebenchmarkprogrammustnotencounteranerrororprematurelyterminate duringorafterthebenchmark. • Thebenchmarkprogrammustbeinausableconditionafterthebenchmark. • Thebenchmarkprogrammaynotdegradeordisabletheuserinterface,orinother mannercauselessworktobedonethanasexpectedbynormaluserinteraction. 1.1TestEnvironment ProcessorIntelPentium4steppingD1 3.2GHzclockspeed 800MHzfrontsidebusspeed Cache8KBL1data 12KBL1instruction 512KBL2unied Memory1GBDDR2700RAM Disk80GBcapacity 100MB/sinterfacebandwidth 49.3MB/saveragetransferrate 19.2msaccesstime OperatingSystemMicrosoftWindowsXPServicePack1a Table1:MachineConguration TheprocessorHyperThreadingextensionwasenabledduringthebenchmarksal- thoughthisdoesnotsignicantlyaffectperformance.Almostallofthebenchmarking workisdoneonasinglethread. Executiontimewasdeterminedbythesystemhighperformancetimer.Thehigh performancetimerhasaresolutionofoneprocessorclockcycle.Theruntimeenvi- ronmentwasconguredtoterminatethebenchmarkprogramifmorethan75%(768 MB)ofphysicalmemorywasallocatedfromtheheap.Duringbenchmarkexecution, thepeakheapallocationsizewasrecorded.Notethatthisvaluemaybemorethan 1

ProgramJigCellversion04302004 CompilerIBMJikesJavacompilerversion1.19 OptimizerSunJavaHotSpotClientVMversion1.4.2.03-b02,mixedmode RuntimeSunJavaRuntimeEnvironmentSEversion1.4.2.03-b02 Table2:SoftwareConguration theamountofmemoryrequiredbythebenchmarkasfreedobjectsneednotbeimme- diatelyreclaimedbytheruntimeenvironmentifmemoryisavilable.Additionally,a lowerboundonthememoryrequiredwasobtainedbydecreasingtheamountofheap memoryavailabletotheruntimeenvironmentuntilthebenchmarkwasunabletocom- pleteduetomemoryexhaustion. 1.2ModelTestData Amodelconsistsofthefollowingelements: • Modelname • Listofcompartments.Eachcompartmenthasanameanddescription. • Listofreactions.Eachreactionhasaname,description,kineticinformation,and parameters.Thekineticequationforrection i is dS d i t +1 =k i ∗ S i − k i+1 ∗ S i+1 startingwith i=0 .Eachparameterhasanameanddescription.Thevalueof parameter i is i+ 1 2 . • Listofspeciesusedinreactions.Eachspecieshasanameanddescription. Thesizeofamodelisdeterminedbythenumberofreactions. 1.3ModelLoadingPerformance ModelloadingperformancemeasuresthesizeofmodelssupportedbytheModelBuilder. Amodelissupportediftheloadingtimeislessthan 300 seconds.TheModelBuilder mustsupportamodelcontaining 1000 reactionstosatisfythemodelbuildingperfor- mancerequirements. Theprocedureformeasuringmodelloadingperformanceis: 1.Createatestmodelcontaining N reactions 2.Writetestmodeltodisk 3.Readtestmodelfromdisk Step3oftheprocedureistimed. AsofMay1st,theJigCellModelBuilderdoesnotpassthisbenchmark.Thetime toloadamodelcontaining 1000 reactionsexceeds 626 secondsusinglessthan 720 MB ofmemoryand 686 KBofdiskspace. 2

1.4ReactionEntryPerformance Reactionentryperformancemeasuresthetimerequiredtoaddanewreactiontothe model.TheModelBuildermustsupportaddingareactiontoamodelcontaining 200 reactionsin 120 secondstosatisfythemodelbuildingperformancerequirements. Theprocedureformeasuringreactionentryperformanceis: 1.Createatestmodelcontaining N reactions 2.Writetestmodeltodisk 3.Readtestmodelfromdisk 4.Addspecies N+1 5.Addreaction N+1 6.Addparameter N+1 7.Writetestmodeltodisk Steps3through7oftheprocedurearetimed. AsofMay1st,theJigCellModelBuilderhasnotbeenrunforthisbenchmark. 1.5RunTestData 1.6RunSetLoadingPerformance Runsetloadingperformancemeasuresthesizeofrunsets,parametersets,andinitial conditionsetssupportedbytheModelBuilder.Asetissupportediftheloadingtime islessthan 300 seconds.TheModelExecutormustsupportarunsetcontaining 10000 runs, 500 parametersets,and 500 initialconditionsetstosatifsythemodelexecution performancerequirements.TheModelExecutordoesnotneedtosupporttheselimits simultaneously. Therearethreeproceduresformeasuringrunsetloadingperformance.Therst proceduremeasuresrunsetloadingperformancewiththemaximumrunsetsize: 1.Createatestmodelcontaining 200 reactions 2.Writetestmodeltodisk 3.Create 1 testparameterset 4.Create 1 testinitialconditionset 5.Createatestrunsetcontaining N runs 6.Writetestrunsettodisk 7.Readtestrunsetfromdisk 3

Thesecondproceduremeasuresrunsetloadingperformancewiththemaximum parametersetsize: 1.Createatestmodelcontaining 200 reactions 2.Writetestmodeltodisk 3.Create N testparametersets 4.Create 1 testinitialconditionset 5.Createatestrunsetcontaining N runs 6.Writetestrunsettodisk 7.Readtestrunsetfromdisk Thethirdproceduremeasuresrunsetloadingperformancewiththemaximumini- tialconditionsetsize: 1.Createatestmodelcontaining 200 reactions 2.Writetestmodeltodisk 3.Create 1 testparameterset 4.Create N testinitialconditionsets 5.Createatestrunsetcontaining N runs 6.Writetestrunsettodisk 7.Readtestrunsetfromdisk Step7ofeachoftheseproceduresistimed. AsofMay1st,theJigCellRunManagerhasnotbeenrunforthesebenchmarks. 1.7RunEntryPerformance Runentryperformancemeasuresthetimerequiredtoaddanewruntoarunset.The ModelExecutormustsupportaddingaruntoarunsetcontaining 2000 runsin 300 secondstosatisfythemodelexecutionperformancerequirements. 1.Createatestmodelcontaining 200 reactions 2.Writetestmodeltodisk 3.Create 100 testparametersets 4.Create 100 testinitialconditionsets 5.Createatestrunsetcontaining N runs 4

6.Writetestrunsettodisk 7.Readtestrunsetfromdisk 8.Addrun N+1 9.Writetestrunsettodisk Steps7through9oftheprocedurearetimed. AsofMay1st,theJigCellRunManagerhasnotbeenrunforthisbenchmark. 1.8RunExecutionPerformance Runexecutionperformancemeasuresthetimerequiredtoexecuteanexistingrun. TheModelExecutormustsupportexecutingarunforaworkingmodelfromarun setcontaining 2000 runsin 60 secondstosatisfythemodelexecutionperformance requirements. Theprocedureformeasuringrunexecutionperformanceis: 1.Createatestmodelcontaining 200 reactions 2.Writetestmodeltodisk 3.Create 100 testparametersets 4.Create 100 testinitialconditionsets 5.Createatestrunsetcontaining N runs 6.Writetestrunsettodisk 7.Readtestrunsetfromdisk 8.Executerun N Steps7and8oftheprocedurearetimed. AsofMay1st,theJigCellRunManagerhasnotbeenrunforthisbenchmark. 1.9RunModicationPerformance Runmodicationperformancemeasuresthetimerequiredtomodifyanexistingrun. TheModelExecutormustsupportmodifyingarunforaworkingmodelfromarun setcontaining 2000 runsin 60 secondstosatisfythemodelexecutionperformance requirements. Theprocedureformeasuringreactionmodicationperformanceis: 1.Createatestmodelcontaining 200 reactions 2.Writetestmodeltodisk 3.Create 100 testparametersets 5

4.Create 100 testinitialconditionsets 5.Createatestrunsetcontaining N runs 6.Writetestrunsettodisk 7.Readtestrunsetfromdisk 8.Settheparameterforreaction 200 inparameterset 100 to 0 9.Settheinitialconditionforspecies 200 ininitialconditionset 100 to 0 10.Writetestrunsettodisk Steps7through10oftheprocedurearetimed. AsofMay1st,theJigCellRunManagerhasnotbeenrunforthisbenchmark. 1.10ExperimentSetTestData Anexperimentsetconsistsofexperiments.Eachexperimenthasthefollowingele- ments: • Experimentname • Globallyuniqueidentier • Experimentalobservations.Theexperimentalvalueisatwovariabletimeseries containing 4 observations.Therstvariableofexperimentobservation i isthe integer i andthesecondvariableisanite,normalizedrealnumber. • Experimentaltype.Thetypemustbecompatiblewiththetimeseriesformatof theexperimentalvalue. • Experimentdescription Thesizeofanexperimentsetisdeterminedbythenumberofexperiments. 1.11ExperimentLoadingPerformance Experimentloadingperformancemeasuresthesizeofexperimentsetssupportedby theModelAnalyzer.Anexperimentsetissupportediftheloadingtimeislessthan 300 seconds.TheModelAnalyzermustsupportanexperimentsetcontaining 10000 experimentstosatisfythemodelanalysisperformancerequirements. Theprocedureformeasuringexperimentloadingperformanceis: 1.Createatestexperimentsetcontaining N experiments 2.Writetestexperimentsettodisk 3.Readtestexperimentsetfromdisk Step3oftheprocedureistimed. AsofMay1st,theJigCellComparatorpassesthisbenchmark.Thetimetoloadan experimentsetcontaining 10000 experimentsislessthan 5 secondsusinglessthan 64 MBofmemoryand 5011 KBofdiskspace. 6

1.12ExperimentEntryPerformance Experimententryperformancemeasuresthetimerequiredtoaddanewexperimentto anexperimentset.TheModelAnalyzermustsupportaddinganexperimenttoanex- perimentsetcontaining 2000 experimentsin 120 secondstosatisfythemodelanalysis performancerequirements. Theprocedureformeasuringexperimententryperformanceis: 1.Createatestexperimentsetcontaining N experiments 2.Writetestexperimentsettodisk 3.Readtestexperimentsetfromdisk 4.Addexperiment N+1 5.Writetestexperimentsettodisk Step3through5oftheprocedureistimed. AsofMay1st,theJigCellComparatorpassesthisbenchmark.Thetimetoadd anewexperimenttoanexperimentsetcontaining 2000 experimentsislessthan 3 secondsusinglessthan 64 MBofmemoryand 998 KBofdiskspace. 1.13TransformSetTestData Atransformsetconsistsoftransformsandamappingfunctionfromexperimentnames totransforms.Eachtransformhasthefollowingelements: • Transformname • Globallyuniqueidentier • Transformationprocedure.Thetransformationprocedureisafunctionalspeci- cationofwhatthetransformdoes. • Parametersusedbythetransformationprocedure • Referencetothetransformthatprovidesinputtothistransform,ifneeded • Listoftransformscontainedinthistransform Theproceduredenedforeachnamedexperimentconsistsoftwotransformsand requiresoneparameter.Thersttransformreceivedinputfromthesecondtransform; thesecondtransformdoesnotrequireinput.Neithertransformcontainsanytrans- forms.Theoutputoftheprocedureisatimeseriesofthesameformatasanexperimen- talobservation.Thesizeofatransformsetisdeterminedbythenumberofmappings fromexperimentnamestotransforms. 7

1.14TransformLoadingPerformance Transformloadingperformancemeasuresthesizeoftransformsetssupportedbythe ModelAnalyzer.Atransformsetissupportediftheloadingtimeislessthan300sec- onds.TheModelAnalyzermustsupportatransformsetfor 10000 experiments,of whichatleast 500 ofthetransformsaredifferent,tosatisfythemodelanalysisperfor- mancerequirements.Whenrunningthisbenchmark,allofthegeneratedtransforms aredifferent. Theprocedureformeasuringloadingloadingperformanceis: 1.Createatestexperimentsetcontaining N experiments 2.Writetestexperimentsettodisk 3.Createatesttransformsetcontaining N transforms 4.Writetesttransformsettodisk 5.Readtestexperimentsetfromdisk 6.Readtesttransformsetfromdisk Step6oftheprocedureistimed. AsofMay1st,theJigCellComparatorpassesthisbenchmark.Thetimetoloada transformsetfor 10000 experimentsislessthan 25 secondsusinglessthan 256 MBof memoryand 15971 KBofdiskspace. 1.15ObjectiveSetTestData Anobjectivesetconsistsofobjectivesandamappingfunctionfromexperimentnames toobjectives.Eachobjectivehasthefollowingelements: • Objectivename • Globallyuniqueidentier • Objectiveprocedure.Theobjectiveprocedureisafunctionalspecicationof whattheobjectivedoes. • Parametersusedbytheobjectiveprocedure Theproceduredenedforeachnamedexperimentrequiresoneparameter.Thesize ofanobjectivesetisdeterminedbythenumberofmappingsfromexperimentnames toobjectives. 8

1.16ObjectiveFunctionLoadingPerformance Objectivefuntionloadingperformancemeasuresthesizeofobjectivesetssupportedby theModelAnalyzer.Anobjectivesetissupportediftheloadingtimeislessthan300 seconds.TheModelAnalyzermustsupportanobjectivesetfor 10000 experiments, ofwhichatleast 50 oftheobjectivefunctionsaredifferent,tosatisfythemodelanal- ysisperformancerequirements.Whenrunningthisbenchmark,allofthegenerated objectivefunctionsaredifferent. Theprocedureformeasuringobjectivefunctionloadingperformanceis: 1.Createatestexperimentsetcontaining N experiments 2.Writetestexperimentsettodisk 3.Createatesttransformsetcontaining N transforms 4.Writetesttransformsettodisk 5.Createatestobjectivesetcontaining N objectives 6.Writetestobjectivesettodisk 7.Readtestexperimentsetfromdisk 8.Readtesttransformsetfromdisk 9.Readtestobjectivesetfromdisk Step9oftheprocedureistimed. AsofMay1st,theJigCellComparatorpassesthisbenchmark.Thetimetoloadan objectivesetfor 10000 experimentsislessthan 10 secondsusinglessthan 256 MBof memoryand 15169 KBofdiskspace. 1.17ExperimentCongurationPerformance Experimentcongurationperformancemeasuresthetimerequiredtocongurethe transformandobjectivefunctionforanewexperiment.TheModelAnalyzermust supportconguringanexperimentinanexperimentsetcontaining 2000 experiments in 300 secondstosatisfythemodelanalysisperformancerequirements. Theprocedureformeasuringexperimentcongurationperformanceis: 1.Createatestexperimentsetcontaining N+1 experiments 2.Writetestexperimentsettodisk 3.Createatesttransformsetcontaining N transforms 4.Writetesttransformsettodisk 5.Createatestobjectivesetcontaining N objectives 9

6.Writetestobjectivesettodisk 7.Readtestexperimentsetfromdisk 8.Readtesttransformsetfromdisk 9.Createtransform N+1 10.Addtransform N+1 toexperiment N+1 11.Writetesttransformsettodisk 12.Readtestobjectivesetfromdisk 13.Createobjective N+1 14.Addobjective N+1 toexperiment N+1 15.Writetestobjectivesettodisk Steps7through16oftheprocedurearetimed. AsofMay1st,theJigCellComparatorpassesthisbenchmark.Thetimetocon- gureanexperimentinanexperimentsetcontaining 2000 experimentsislessthan 20 secondsusinglessthan 256 MBofmemoryand 5207 KBofdiskspace. 1.18SummaryofResults AsofMay1st,6ofthe13benchmarkshavebeencompletedandrun.JigCellpasses5 ofthebenchmarksandfails1ofthebenchmarks. #Time(s)Memory(MB)Disk(KB)Passes 1.3626.28512-720686 4.1 a6.1 b6.1 c6.1 7.1 8.1 9.1 1.115.0032-495011 • 1.122.8932-41998 • 1.1424.03128-18715971 • 1.169.94128-15015169 • 1.1719.4532-2565207 • Table3:BenchmarkResultsSummary 01

Figure Figure :1 :2 Benchmark Benchmark 11 3.1 11.1