37 pages

Slovak

G-Lite Application Development Tutorial

Orku - Oliver Malham

Le téléchargement nécessite un accès à la bibliothèque YouScribe
Tout savoir sur nos offres

37 pages

Slovak

Le téléchargement nécessite un accès à la bibliothèque YouScribe
Tout savoir sur nos offres

A propos
Informations
Extrait

Description

gLite A pplication De velopment Tut orialWritten b y Oli ver M alham, Sar a Colli nsCode writt en b y Sar a Colli nsNati onal e -Science C entre, UK(C orrespondence to oliver@ nesc. ac.uk ) ContentsIntroduction...................................................................................... 5Motivation..................................................................................... 5Document L ayout.......................................................................... 5Documentation Sources................................................................ 6C++ S tandard T emplate Library (S TL) ....................................... 6RGMA......................................................................................... 6Fireman 6gLite I/O 7Workload Ma nagement System (W MS) ...................................... 71 RGMA............................................................................................. 81.1 Your First Qu ery...................................................................... 81.1.1 Co nsumerFactory o bject................................................... 81.1.2 Timeout............................................................................ 81.1.3 Co nsumer......................................................................... 91.1.4 La unch t he query.............................................................. 91.1.5 W ait........................................................................ ...

Informations

Publié par	Orku
Nombre de lectures	64
Langue	Slovak

Extrait

gLite Application Development Tutorial

Written by Oliver Malham, Sara Collins Code written by Sara Collins

National e-Science Centre, UK (Correspondence tokucsen.ca.oliver@)

Contents

Introduction......................................................................................5 Motivation.....................................................................................5 Document Layout..........................................................................5 Documentation Sources................................................................ 6 C++ Standard Template Library (STL)....................................... 6 RGMA.........................................................................................6 Fireman.....................................................................................6 gLite I/O..................................................................................... 7 Workload Management System (WMS)...................................... 7 1 RGMA.............................................................................................8 1.1 Your First Query...................................................................... 8 1.1.1 ConsumerFactory object................................................... 8 1.1.2 Timeout............................................................................8 1.1.3 Consumer.........................................................................9 1.1.4 Launch the query.............................................................. 9 1.1.5 Wait..................................................................................9 1.1.6 Retrieve the results........................................................ 10 1.1.7 Iterate............................................................................. 10 1.1.8 Retreive the metadata.................................................... 10 1.1.9 Iterate over the metadata............................................... 11 1.1.10 Column name and type................................................. 11 1.1.11 Data.............................................................................. 11 1.1.12 Clean up....................................................................... 12 1.1.13 Error handling............................................................... 12 1.1.14 Destroy the factory....................................................... 12 1.1.15 Compile........................................................................ 13 1.1.16 Run...............................................................................13 1.2 Publish and Consume Data.................................................... 13 1.2.1 Producer Factory............................................................ 13 1.2.2 Create a timeout............................................................. 14 1.2.3 Producer Properties........................................................ 14 1.2.4 Create a primary producer.............................................. 14 1.2.5 Data retention................................................................15 1.2.6 Declare the table............................................................ 15 1.2.7 Insert the data................................................................ 15 1.2.8 Producer Clean-up.......................................................... 15 1.2.9 Handle Errors.................................................................. 16 1.2.10 Continuous Consumer................................................... 16 1.2.11 Compile........................................................................ 16 1.2.12 Run...............................................................................17 1.3 Consume Historical Data....................................................... 17 1.3.1 Create Consumer............................................................ 17 1.3.2 Compile and Run............................................................ 18

1.4 Secondary Producer.............................................................. 18 1.4.1 Extended Time-out......................................................... 18 1.4.2 Secondary Producer........................................................ 18 1.4.3 Data Retention .............................................................. 19 1.4.4 Declare the table............................................................ 19 1.4.5 Keep alive....................................................................... 19 1.4.6 Catch exceptions............................................................ 19 1.4.7 The Client: Resource list................................................. 20 1.4.8 Start the consumer......................................................... 21 1.4.9 Compile.......................................................................... 21 1.4.10 Run...............................................................................21 2 Fireman........................................................................................ 21 2.1 Directory Creation................................................................. 22 2.1.1 Catalogue context.......................................................... 22 2.1.2 Error Handling................................................................. 22 2.1.3 Endpoint.........................................................................22 2.1.4 Create a directory........................................................... 22 2.1.5 cleanup........................................................................... 23 2.1.6 Compile and Run............................................................ 23 2.2 List Directory......................................................................... 23 2.2.1 Create a context............................................................. 23 2.2.2 Display the endpoint....................................................... 23 2.2.3 Check the permission to read the directory.................... 23 2.2.4 Catalog list...................................................................... 24 2.2.5 Display............................................................................ 24 2.2.6 Cleanup..........................................................................24 2.1.6 Compile and Run............................................................ 24 3 gLite-IO........................................................................................ 25 3.1 Write a File............................................................................ 25 3.1.1 Open the file................................................................... 25 3.1.2 Write the content............................................................ 25 3.1.3 Close the file................................................................... 26 3.1.4 Compile and Run............................................................ 26 3.2 Read a File............................................................................ 26 3.2.1 Open the file for reading................................................. 26 3.2.2 Read the data................................................................. 26 3.2.3 Cleanup..........................................................................27 3.2.4 Compile and Run............................................................ 27 3.3 Read a File Segment............................................................. 27 3.3.1 Starting offset................................................................. 27 3.3.2 Move the filepointer........................................................ 27 3.3.3 Compile and Run............................................................ 28 4 Fireman........................................................................................ 29 4.1 Locate a File.......................................................................... 29 4.1.1 Context........................................................................... 29 4.1.2 Get the list of filenames that match the lfn and pattern. 29 4.1.3 Cleanup..........................................................................29 4.1.4 Compile and Run............................................................ 29 4.2 Catalogue Metadata.............................................................. 30

4.2.1 List replicas.................................................................... 30 4.2.2 Display the LFN and GUID for the returned catalogue entry........................................................................................30 4.2.3 Get the permissions for the entry................................... 30 4.2.4 Display the permissions.................................................. 31 4.2.5 Display the URLs of all the replicas for the given LFN..... 31 4.2.6 Compile and Run............................................................ 31 5 Workload Management System (WMS)........................................ 31 5.1 Job Submission and Monitoring.............................................. 32 5.1.1 Declare a JobAd.............................................................. 32 5.1.2 Set the executable attribute........................................... 32 5.1.3 Configure the Job............................................................ 32 5.1.4: Add the Output Sandbox................................................ 33 5.1.5 Add the Rank.................................................................. 33 5.1.6 Add the requirements..................................................... 34 5.1.7 Check the JDL................................................................. 34 5.1.8 Create a job object......................................................... 34 5.1.9 Logging level.................................................................. 34 5.1.10 Submit the job.............................................................. 35 5.1.11 Display the jobID.......................................................... 35 5.1.12 Exceptions.................................................................... 35 5.1.13 Create a JobId............................................................... 35 5.1.14 Create a Job.................................................................. 36 5.1.15 Check job progress....................................................... 36 5.1.16 Extract logging info....................................................... 36 5.1.17 Print out logging info.................................................... 36 5.1.18 Get the output of the job.............................................. 37 5.1.19 Compile and Run.......................................................... 37

Introduction

Motivation The goal of this tutorial is to demonstrate the various APIs that you need to use in order to write a gLite enabled grid application. We will focus on resource discovery, file handling, and remote job execution and management. The primary purpose of this tutorial is not to tell you line for line what your code should be, as this teaches you nothing except how to copy. Rather, we feel that the most useful skill that we can teach you today is how to find the information you need to use gLite to write your own code for your own projects. With that in mind, all of the practical exercises consist of a skeleton program and a description of what it should do, with references to the API documentation so that you can add the required functionality.

Document Layout The names of source files, classes and structures will have this formatting: ClassName. Documentation references will be highlighted thus: Example reference: On-line course material for GridKa http://homepages.nesc.ac.uk/~sara/gridka/ Commands that must be entered in the shell exactly as shown, are highlighted in a similar fashion: $ example command 1 _ _ $ example command_2 _ Before you can compile or run any of the code in this tutorial, you must first logon to the GILDA UI, glite-tutor and start a grid proxy (entering the appropriate pass-phrase to do so). GILDA can only be accessed from the GridKa machine gks02, gks03 and gks04. These are the commands that you will have to run: from a GridKa machine $ ssh –p 22 glite-tutor.ct.infn.it –l karlsruhenn wherenncan be a value between 02 and 79. The password for the accounts are GridKARnnwhere thennmust match the value used for the username. Once on glite-tutor create a proxy using $ grid-proxy-init with the passphrase KARLSRUHE. You now need to obtain the skeleton code for the tutorial. This is available on the tutorial web-page given above. Use the commands $ wget http://homepages.nesc.ac.uk/~sara/gridka/code.tar.gz $ tar zxvf code.tar.gz

to transfer the code to your home directory on glite-tutor and unpack the tarball. The skeleton code is now sitting in the directorygridka within subdirectories for each stage of the tutorial. The instructors will have model solutions, so please ask for help if you get stuck. In addition, there is example output from each of the exercise programs on the tutorial web pages atirg//akd~/kuarasnesc.ac.mepages.tt:p//ohh. Documentation Sources This tutorial will make heavy use of the user guides and API reference material for the various gLite components. Here is a list of links to the documents that we will be using during this tutorial (each tutorial section has more detailed references). C++ Standard Template Library (STL) ·API Reference http://www.sgi.com/tech/stl/table_of_contents.html Note: This is not a gLite library, but a standard library of C++ templates and classes that some of the other gLite libraries make use of. For those of you not experienced with C++, the STL may be an unknown, hence the API reference. RGMA ·C++ User Guide http://hepunx.rl.ac.uk/egee/jra1-uk/glite-r1/cpp.pdf ·C++ API Reference http://hepunx.rl.ac.uk/egee/jra1-uk/glite-r1/cpp-api/hierarchy.html

Fireman ·User guide https://edms.cern.ch/file/570780/1/EGEE-TECH-570780-C-CPP-API-v1.0.pdf ·API Reference https://edms.cern.ch/file/570780/1/EGEE-TECH-570780-C-SIMPLE-API-v1.0.pdf

gLite I/O

·User guide and API reference https://edms.cern.ch/file/570771/1/EGEE-TECH-570771-v1.0.pdf

Workload Management System (WMS)

C++ and Java API references

http://egee-jra1-wm.mi.infn.it/egee-jra1-wm/glite-wms-api-index.shtml

Users guide

https://edms.cern.ch/file/572489/1/WMS-guide.pdf

1 RGMA

In this section of the tutorial, you will start to get to grips with the RGMA information system. All of the files for this section are located in thestage1 rgma directory in _ your tutorial account. All documentation references refer to the RGMA API web pages: http://hepunx.rl.ac.uk/egee/jra1-uk/glite-r1/cpp-api/hierarchy.html

If you find that you spend a lot of time completing section 1.1 we suggest you skip 1.2-1.4 and go on to section 2 so that you have time to complete an exercise from each section. 1.1 Your First Query The first program that you are going to write that works with the RGMA system is ConsumerLatest.cpp. This program will allow you to perform SQL queries on the system, returning the latest results. It will take the SQL query from the command line, submit it to the RGMA service (the API automatically chooses which producer to use), and wait for the result to be returned. Once the result is received, it will be displayed on the console for you to see.

1.1.1 ConsumerFactory object The first thing that your code should do, after checking that it received an SQL query on the command line, is to create theConsumerFactoryobject. This will be used to manage theConsumerobject that will dispatch the query and handle the results. You need to create a local variable calledfactory, which is a ConsumerFactoryYour code should create a newpointer. ConsumerFactoryImpl, and assign its pointer to thefactorylocal variable. This should be created immediately after the comment labelled1.1.1. ConsumerFactory reference http://hepunx.rl.ac.uk/egee/jra1-uk/glite-r1/cpp-api/classglite_1_1rgma_1_1ConsumerFactory.html

1.1.2 Timeout Before the Consumer object can be created, we need to define a time interval that will be used to limit the time that the query is permitted to run. This is to prevent network disconnections, orphaned processes, etc. from tying up finite resources. The time interval is defined by creating aTimeIntervalobject. For our purposes a time limit of one minute should suffice. You should add code to create a local variable calledterminationInterval, of typeTimeInterval, and initialise it with a value of one minute. This should be added after the comment labelled1.1.2. TimeInterval reference

http://hepunx.rl.ac.uk/egee/jra1-uk/glite-r1/cpp-api/classglite_1_1rgma_1_1TimeInterval.html Units reference http://hepunx.rl.ac.uk/egee/jra1-uk/glite-r1/cpp-api/classglite_1_1rg _ _ .html ma 1 1Units

1.1.3 Consumer Now our program needs to create theConsumerobject that will dispatch the query to the Consumer Service (which contacts the RGMA registry and relevant Producers). Consumerobjects are created and initialised by theConsumerFactoryobject that we created in section 1.1.1. You need to add a line of code that creates a local variable calledconsumer, aConsumerclass pointer, and initialise it with the result of a call to thecreateConsumermethod of the class referred to by thefactory pointer. The query mode should beLATEST, the timeout should be timeIntervalfrom the command line. Add this, and the query should be taken code after the comment labelled1.1.3. ConsumerFactory reference http://hepunx.rl.ac.uk/egee/jra1-uk/glite-r1/cpp-api/classglite_1_1rgma_1_1ConsumerFactory.html QueryProperties reference http://hepunx.rl.ac.uk/egee/jra1-uk/glite-r1/cpp-api/classglite_1_1rgma_1_1QueryProperties.html

1.1.4 Launch the query The program should now have everything that it needs in order to launch the query. Use thestartmethod of theconsumervariable to submit the query to RGMA. Make sure that you give thestartmethod a time limit of one minute, so the API will not wait forever for a failed request. This code should be entered after the comment labelled1.1.4. Consumer reference http://hepunx.rl.ac.uk/egee/jra1-uk/glite-r1/cpp-api/classglite_1_1rgma_1_1Consumer.html

1.1.5 Wait Now that the query has been launched, our program should wait for it to finish executing. You should add a loop after the comment labelled1.1.5which repeatedly calls thesleepinterval of five seconds while thesystem call with an consumerreports that it is still executing. Consumer reference

http://hepunx.rl.ac.uk/egee/jra1-uk/glite-r1/cpp-api/classg _ _ g _1_1Consumer.html lite 1 1r ma

1.1.6 Retrieve the results Assuming no errors have occurred (covered by section 1.1.13), ourconsumer variable should now be in possession of a result set. However, before we can do anything with the results, we need to extract them from theConsumerobject. After the comment labelled1.1.6, you code should create a local variable called resultSet, of typeResultSet, and then (on the following line), use the popAllmethod of theConsumerclass to extract the results intoresultSet. ResultSet reference http://hepunx.rl.ac.uk/egee/jra1-uk/glite-r1/cpp-api/classglite_1_1rgma 1 1ResultSet.html _ _ Consumer reference http://hepunx.rl.ac.uk/egee/jra1-uk/glite-r1/cpp-api/classglite_1_1rgma_1_1Consumer.html

1.1.7 Iterate Because theConsumercould potentially return multiple sets of results, we need to iterate over the set of results stored inresultSet. We do this using aforloop and aResultSet::iteratorobject. Yourforloop, after the comment labelled 1.1.7, should create a new variable calledcurrentResult, and initialise currentResultusing thebeginmethod of theresultSetvariable. The loop should end when the iterator equals the result of theendmethod of theresultSet variable, and it should be incremented once every iteration. The set of results in resultSet typeis a vector ofTupleso thatcurrentResultis a pointer to an object of typeTuple. ResultSet reference http://hepunx.rl.ac.uk/egee/jra1-uk/glite-r1/cpp-api/classglite_1_1rgma_1_1ResultSet.html

1.1.8 Retreive the metadata Each set of results (i.e.Tuple) contains more than just the raw data. It also includes the name and datatype of each column in that result set, the “metadata”. Your program needs the metadata to determine how to retrieve the data, and we will display the column name to give the data some context for the user. The first step is to retrieve the metadata from thecurrentResultiterator. After the comment labelled 1.1.8, create a local variable calledmetaData, which is a pointer to a ResultSetMetaDatainitialised with the return value of theobject. This should be getMetaDatamethod of theTuple class applied to thecurrentResult object (remembercurrentResultis a pointer to an object of typeTuple). ResultSet reference

http://hepunx.rl.ac.uk/egee/jra1-uk/glite-r1/cpp-api/classglite_1_1rgma_1_1Tuple.html ResultSetMetaData reference http://hepunx.rl.ac.uk/egee/jra1-uk/glite-r1/cpp-api/classglite_1_1rgma_1 1ResultSetMetaData.html _

1.1.9 Iterate over the metadata Now that we have the metadata, we need to iterate over it so that we can print out the column name and value for each column in the tuple. As in section 1.1.7, we will use aforloop, and create, initialise and increment an iterator object. You should call this objectcolumn, and it is aResultSetMetaDataiterator. Your loop should continue for as long ascolumnis not equal to the end of themetaDataobject. You should fill out the emptyforloop statement in your code under the comment labelled 1.1.9. ResultSetMetaData reference http://hepunx.rl.ac.uk/egee/jra1-uk/glite-r1/cpp-api/classglite_1_1rgma_1_1ResultSetMetaData.html

1.1.10 Column name and type We are now ready to extract the column name and type from the current column. The column name we will print out on the console for the user, the datatype we will use in a moment to determine how to retrieve and display the data itself. Below the comment labelled1.1.10, create a new local variable calledcolumnNameof typestring. This should be initialised to the name of the column, using a method call of the ResultSetMetaDataobject. On the line after this, you should create another local variable, calledcolumnType, of typeint. Again, this should be initialised using the appropriateResultSetMetaDatamethod call. Finally, the column name should be printed out on the console as a label for the data that we will retrieve and display in the next section. ResultSetMetaData reference http://hepunx.rl.ac.uk/egee/jra1-uk/glite-r1/cpp-api/classglite_1_1rgma_1_1ResultSetMetaData.html

1.1.11 Data Finally, we are ready to extract and display the data. Under the comment labelled 1.1.11, you should add a set of fiveifstatements comparing the value of columnTypeagainst the constants defined by theTypesclass in order to determine the datatype of the current column. The body of eachifstatement should retrieve fromcurrentResultentry for the current column in the currentthe value of the result set (tuple), using the accessor functions in theTupleclass. You should then print out the value of this data, followed by a new line character so the user can easily read the query results.