Report on the international seminar New Developments in Performance Audit – Methods and Processes

Informations

• Publié par: Ziom
• Nombre de lectures: 47
• Langue: English

Extrait

 

 

 

                   

    Report on the international seminar New Developments in Performance Audit – Methods and Processes 10-11 May 2005      

The National Audit Office of Denmark August 2005

 

 

 

 

 

 

Contents

Page  Programme for the seminar New Developments in Performance Audit – Methods and Processes ............................................................................... 5 List of participants ........................................................................................ 7 Theme A: Different Approaches to Quality Assurance........................... 8 Development of a Comprehensive Quality Assurance Framework at the UK National Audit Office (NAO).................................................................... 8 The Development of a New Quality Assurance Model ................................. 9 One-Pass Planning (OPP).......................................................................... 10 The Use of Qualitative Data as Evidence in Performance Auditing (The OAG of Norway) ......................................................................................... 11 Theme B: Challenges in the Use of Methods in Performance Auditing..................................................................................................... 13 Primary Data Analysis ................................................................................ 13 The Use of Computer Assisted Audit Techniques (CAATS) during the Planning and Execution of Performance Audits ......................................... 14 Lessons Learned in Using Performance Audit Methods – Toolbox ............. 15 Theme C: Future Challenges for SAIs in Performance Audit............... 17 Challenges in Performance Auditing of Aid Management and Evaluation of Overseas Aid Programmes .................................................................... 17 Evaluating Weapon System Programs Using a Knowledge-Based Approach .................................................................................................... 18 Future Perspectives for Effective Performance Auditing: the Role of the Supreme Chamber of Control of Poland (SCC) in Improving Public Funds Managements and Presentation of Performance ............................ 19 Evolving Arrangements to Achieve Financial Impact from Performance Audit Work at the UK National Audit Office (NAO) ..................................... 20 Performance Gap-focused Research: Looking for Explanations................ 21 Overall summary...................................................................................... 23 Close of the seminar and next venue..................................................... 24

 

 

 

The National Audit Office of Denmark

Programme for the seminar New Developments in Performance Audit – Methods and Processes Monday 9 May 2005 20.00 Reception at Phoenix Hotel hosted by the National Audit Office of Denmark. The reception will take place in room Gyldensteen.     Tuesday 10 May 2005   08.45 Pick up at Phonix Hotel by Bjørn Olsen. 09.00Welcome address by the National Audit Office of Denmark. The seminar will take place at the National Audit Office of Denmark, St. Kongensgade 45, in the Green Meeting Room 3 rd floor. 09.15 Delegates briefly introduce themselves. 09.30 Session 1 A: Different approaches to quality assurance The National Audit Office of United Kingdom and Sweden will do a 15 minutes for- mal presentation on each of their papers, which will be followed by questions for clarification.  Ms. Frances Taylor from the National Audit Office of Canada will facilitate the session. 10.15 Break. 10.30 Session 1 B: Different approaches to quality assurance– continued The National Audit Office of Canada and Norway will do a 15 minutes formal pres- entation on each of their papers, which will be followed by questions for clarification. Ms. Frances Taylor from the National Audit Office of Canada will facilitate the session. 11.15 Break. 11.30 Session 1 C: Different approaches to quality assurance– continued Ms. Frances Taylor from the National Audit Office of Canada will lead the informal round table discussion.   Wrap-up of session 1 A-B by facilitator Ms. Frances Taylor. 12.30 Lunch. 13.30 Session 2 A: Challenges in the use of methods in performance audit The National Audit Office of Ireland and the Republic of South Africa will do a 15 minutes formal presentation on each of their papers, which will be followed by ques- tions for clarification.   Mr. Peter van der Knaap from the National Audit Office of the Netherlands will facili- tate the session. 14.15 Break.   14.30 Session 2 B: Challenges in the use of methods in performance audit– continued The National Audit Office of Denmark will do a 15 minutes formal presentation, which will be followed by questions for clarification.   Mr. Peter van der Knaap from the National Audit Office of the Netherlands will facili- tate the session.   15.00 Break.      

 

Page 5

The National Audit Office of Denmark

  15.15 Session 2 C: Challenges in the use of methods in performance audit– continued Mr. Peter van der Knaap from the National Audit Office of the Netherlands will lead the informal round table discussion.   Wrap-up of session 2 A-B by facilitator Mr. Peter van der Knaap.   16.15 Close of the first day of the seminar.   18.30 Pick-up by bus at Phoenix Hotel.   19.00-22.00 Dinner in Tivoli at Restaurant Paafuglen (telephone [phone]) hosted by the National Audit Office of Denmark.     Wednesday 11 May 2005 09.00 Session 3 A: Future challenges for SAIs in performance audit The National Audit Office of Australia and the United States will do a 15 minutes formal presentation on each of their papers, which will be followed by questions for clarifi- cation.   Mr. Paul Francis from GAO of the United States will facilitate the session.   09.45 Break.   10.00 Session 3 B: Future challenges for SAIs in performance audit– continued The National Audit Office of Poland, United Kingdom and the Netherlands will do a 15 minutes formal presentation on each of their papers, which will be followed by questions for clarification.   Mr. Paul Francis from GAO of the United States will facilitate the session.   11.00Break.   11.15 Session 3 C: Future challenges for SAIs in performance audit– continued Mr. Paul Francis from the GAO of the USA will lead the informal round table discus- sion.   Wrap-up of session 3 A-B by facilitator Mr. Paul Francis.   12.15 Close of the seminar and next venue.   12.30-13.30 Farewell lunch hosted by the National Audit Office of Denmark   The lunch will take place at Restaurant M, St. Kongensgade 56 (telephone [phone]).    

Page 6

 

The National Audit Office of Denmark

List of participants   Country SAI E-mail-adress Group Executive Director Australia Australian National Audit [e-mail] John Meert Office Principal Frances Taylor Canada Office of the Auditor [e-mail] General of Canada Director Hans Andersen Denmark National Audit Office of [e-mail] Denmark Director Bjørn Olsen Denmark National Audit Office of [e-mail] Denmark Head of Section Denmark National Audit Office of [e-mail] Thomas Førby Sørensen Denmark Head of Section Denmark National Audit Office of [e-mail] Lars Østergaard Denmark Head of Division ECA European Court of Audi- [e-mail] Eduardo Ruiz Garcia tors, ADAR Dr. Elisabeth Türk ECA European Court of Audi- [e-mail]. tors, Professional Trainin Unit Chief Auditor Olav Lüüs Estonia State Audit Office of [e-mail] Estonia Senior Auditor Estonia State Audit Office of [e-mail] Janno Veskimets Estonia Senior Auditor Ireland Office of the Comptroller [e-mail] Patricia Sheehan and Auditor General of Ireland Dr. Gert Jan De Vries Netherlands Netherlands Court of Audit Director/Dr. Netherlands Netherlands Court of Audit [e-mail] Peter van der Knaap Senior Audit Adviser Norway National Audit office of jens-are-enoksen Jens-Are Enoksen Norway @riksrevisjonen.no Deputy Director General Norway National Audit office of [e-mail] Trond Hjelle Norway Director General Norway National Audit office of [e-mail] Therese Johnsen Norway Senior Inspector Poland Supreme Chamber of Monika [e-mail]. Monika Gruszeczka Control – Poland Main S ecialist Poland Su reme Chamber of Andrzej [e-mail] Andrzej Pogoda Control – Poland Main Specialist Poland Supreme Chamber of Jacek [e-mail] Jacek Stankiewicz Control – Poland Senior Manager South Africa Office of the Audito [e-mail] Christina Laubscher General of South Africa Director Inge Danielsson Sweden The Swedish National [e-mail] Audit Office Audit Director Sweden The Swedish National ingemar.delveborn @riksrevisionen.se Ingemar Delveborn Audit Office Mr. Jeremy Lonsdale United National Audit Office, UK Kingdom Director Robert Prideaux United National Audit office, UK [e-mail] Kingdom Assistant Auditor General United National Audit Office, UK [e-mail] Michael   Whitehouse Kingdom Director Paul Francis USA Government [e-mail] Accountability Office  

 

Page 7

The National Audit Office of Denmark

Theme A: Different Approaches to Quality Assurance Development of a Comprehensive Quality Assurance Framework at the UK National Audit Office (NAO) Mr. Michael Whitehouse, Assistant Auditor General, NAO of the UK   Presentation of the NAO total quality assurance frame- work covering all phases of an audit including ‘on-the-job’ quality management, post project review, communication strategy and ongoing strategic measures to improve qual- ity assurance The objective of the framework is to secure that audit reports are delivered in time and at the agreed cost and quality. It is structured around five quality thresh- olds which the PA group can only pass having considered and affirmed some key questions. Presentation of future areas for development, including development of a quality manual, further development of methodological frame- work, and enhancement of technical guidance and train- ing.   Main issues:   • risks to quality in PA Controlling • management Project • 5 quality thresholds  How to develop a framework for quality assurance •  Strategic planning of QA •   Discussion   The NAO approach is highly structured and developed on a sound basis over many years. The approach seeks to ensure a balance between bureaucracy and creativity in managing risks. The presentation was related to the vari- ous PA studies undertaken and did not cover the quality of the overall planning and risk assessment of NAO. As for all SAI’s, there is an inherent danger of ‘missing the target’, when questions continually arise in the public, which have not been addressed in reporting. We may hit a small, but risky area, but miss the major issue right next to it. The questions asked are e.g. whether we are able to combine our risk assessments across the organisation to take all available knowledge into account.   See the presentation by Mr. Michael Whitehouse

Page 8

 

 

The National Audit Office of Denmark

The Development of a New Quality Assurance Model Mr. Inge Danielsson, Director of the Quality and Methods Division, and Mr. Ingemar Delveborn, Audit Director, the Swedish NAO   Presentation of experiences from developing and imple- menting a new quality assurance model in a relatively new integrated SAI. The SAI had got a new mandate and had to establish a complete new set of guidelines includ- ing a new QA-model with five critical milestones. The staff coming from the previous organisations had, however, a long PA experience. The SAI was organized as a line or- ganization with six divisions for PA. Each Head of Division had the overall responsibility for meeting the milestones of the QA-model. A second opinion was given through the independent QA-system.   Main Issues:   • background for developing a QA model Organisational • The objectives of a QA system • Independent quality review   Discussion   As for the NAO(UK), the QA system of the Swedish Na- tional Audit Office is highly structured, but in addition to this it has been assigned an important and very unique role within the new organization. It was recognized that the quite flat structure of the organization and the high AG interaction demanded a supplementary and independent quality review department. A well staffed QA department could be regarded as a necessary condition for the re- moval of an entire management level, but it did involve, however, some bureaucratic problems.   In the discussion there was some concern that we may all be focussing too much on readability as opposed to whether we do the right thing, using the appropriate meth- odologies. In particular, there were some reflections on whether we all had the right balance between the time used for the study until the first draft and the subsequent time used for QA and final report preparation.   See the presentation by Mr. Inge Danielsson and Mr. In- gemar Delveborn  

Page 9

The National Audit Office of Denmark

One-Pass Planning (OPP) Ms. Frances Taylor, Principal, Canada   The OPP was presented as a strategic planning and deci- sion-making tool for documenting and assessing risks. The process was introduced in 2002 and the risk descrip- tion and profile will be up-dated every 3-4 years. It re- quired an extensive planning, i.e. each analysis would take about 600 hours. Introduction of an integrated audit framework, including risk analysis, OPP reporting, and at- test audit. The relevant type of audit is stated for the vari- ous elements in the risk analysis. The presentation focus- sed on the advantages and principles of OPP in risk man- agement and planning.   Main Issues:   • of risk and risk management Definition • Strategic risk assessment and planning • of risk analysis Documentation   Discussion   The Canadian approach integrates risk analysis for regu- larity (attest) and performance audit. At the end of the process the analysis is checked against the risk assess- ments carried out by the administration, which is critically evaluated by the OAG. The approach led to a large num- ber of potential performance reviews the Canadian SAI could undertake. The OPP was regarded as a very strong planning and risk analysis tool, although, senior manage- ment still needs to establish the relative priorities between reviews and between the various sectors. There was also a clear risk involved for the OAG – and the Government entity - if the many potential audits identified through the OPP process were not subsequently audited.   The discussion showed in general that all SAI’s to a lesser or greater extent should be able to justify their se- lection of PA’s and to answer questions like “where was the auditor”.   See the presentation by Ms. Frances Taylor

Page 10

 

 

The National Audit Office of Denmark

The Use of Qualitative Data as Evidence in Performance Auditing (The OAG of Norway) Mr. Jens-Are Enoksen, Senior Audit Advisor, Norway   The analysis of causal relationships can often benefit from using qualitative analytical tools. How to ensure the qual- ity of evidence and documentation in PA. Introduction to the use of qualitative data in PA and methodological tools to improve the potential of qualitative documentation. Presentation of a PA case on organisational explanations to differences in efficiency in Norwegian hospital. Use of the software package Nvivo as a means of processing in- terview material and documenting and assuring the qual- ity of the analysis and lessons learned.   Main Issues:   • Assuring data quality and documentation • Case on organisational explanations to differences in ef- ficiency • to improve potential of qualitative data Methods •Computer Assisted Audit Techniques     Discussion   It was recognized that in order to improve the manage- ment of an audited department it was important for the auditor to understand the causes of system risks. This can often best be done by using qualitative data. Accord- ingly, the QA of qualitative data becomes important. Fun- damentally, there is nothing new in using a CAAT system to evaluate qualitative data, compared to more traditional methods. The use of the particular system does take some time and resources to set up properly, so there needs to be staff specialization and/or reasonable large amount of data to process to make it worthwhile. When used, it is a strong analytical tool and gives increased confidence in findings. It also facilitates the QA process as the documentation becomes more transparent and readily available.   See the presentation by Mr. Jens-Are Enoksen  

Page 11