Systems Risk Factors and Audit Planning

Jean Bedard

Sujets

• management
• risk management
• managers

Informations

• Publié par: Anjo
• Nombre de lectures: 23
• Langue: English

Extrait

INFORMATION SYSTEMS RISK FACTORS, RISK ASSESSMENTS, AND AUDIT PLANNING DECISIONS                 Jean C. Bedard   Cynthia Jackson   Both at: College of Business Administration 404 Hayden Hall Northeastern University Boston, MA 02115       Lynford Graham Director of Audit Policy BDO Seidman LLP 330 Madison Avenue 10th Floor New York, NY 10017

        Acknowledgments: The authors thank the public accounting firms providing advice and the time of their personnel in support of this research. We also appreciate helpful comments from Kathy Hurtt, Ganesh Krishnamoorthy, and Margarita Lenk.  

 

INFORMATION SYSTEMS RISK FACTORS, RISK ASSESSMENTS, AND AUDIT PLANNING DECISIONS     ABSTRACT   In this study, we examine systems risk factors identified by external auditors for a sample

of their actual audit clients. Specifically, we study two important areas of information systems

risk: the risk of breaches in system security and the risk that the information provided by the

system is inadequate. To perform the study, we examine the nature of systems risk factors

identified, and relate those risk factors to the auditors systems risk assessments and audit test

plans. We find that systems risk factors are identified for a high proportion of clients, most

frequently including issues of management style and competence, maintaining system currency,

and adequacy of documentation. Risk assessments are significantly associated with the number

of risk factors identified within each area, and more risk factors are identified in clients with

higher business risk. To address systems risk factors, auditors most often choose review/inquiry

procedures. To a lesser extent, we observe the design of some tests of controls to address EDP

security risk issues and the design of some substantive testing procedures to address management

information risk issues. Audit test planning is statistically associated with system-specific types

of risk factors in the EDP security risk area, and with both company-level and system-specific

risk factors in the management information risk area.

 

Key words:Information systems, Control risk, Systems risk, Audit risk, Audit planning

INFORMATION SYSTEMS RISK FACTORS, RISK ASSESSMENTS, AND AUDIT PLANNING DECISIONS     INTRODUCTION   The purpose of this study is to examine external auditors perspectives on information systems risk in their actual audit clients. During the past two decades, companies have invested

considerable resources in information technology (IT). These organizations rely on IT to collect,

maintain and communicate data to support achievement of their objectives, and to measure their

achievements both internally and externally (e.g., Tucker 2001). When developing IT systems,

enterprises tend to focus on the benefits of technology. However, they should also recognize the

need to understand and manage its associated risks. The greater the reliance on IT for managing

and controlling key business operations, the greater the likelihood that inadequate systems will

prevent business goals from being achieved. While research on systems risk is important in

gaining an understanding of how systems can be improved, there is little evidence in the

literature regarding this issue. This study uses auditors evaluations of their clients systems risk factors as a source of

data on the difficulties frequently seen in company systems. In considering a clients IT

environment, auditors need to understand the risk that systems may not perform as planned. This

implies that specific risk factors related to IT functioning will be identified and documented, and

their potential role in the engagement will be analyzed along with other elements of the internal

control environment. The Auditing Standards Board has recently focused attention on the role of

clients information systems in controlling business processes in SAS No. 94, The Effect Of

Information Technology On The Auditors Consideration Of Internal Control In A Financial Statement Audit (AICPA 2001). This new standard particularly emphasizes that the auditor should obtain an accurate assessment of the role of a clients systems in its internal control

2    environment, including both the quality of the systems and the functions performed by them. Additionally, the Sarbanes-Oxley Act of 2002 (H.R. 3763) will further emphasize internal controls. The Act will require public companies to file a report on internal controls with their annual reports. This report will include managements assessment of the effectiveness of control design and procedures, and the companys auditors will report on this assessment. In addition to its importance for auditors, research on auditors consideration of IT risk should also interest corporate systems professionals. The auditors evaluation of a clients systems risk is an important source of information on systems quality. The systems literature has incurred difficulty in evaluating systems effectiveness (Stone 1990; Arnold 1995). One means of gathering information on systems effectiveness (within a particular companys system or across companies) is to employ information from auditors. Because they consider systems risks for a variety of clients during every audit cycle, auditors are in a unique position to identify weaknesses that inhibit system performance. Once risk factors have been identified and defined, solutions can be developed, implemented and integrated into the business process. Thus, research on factors that influence auditors views of their clients systems is of potential benefit both to management and systems developers in improving systems quality. Armed with this knowledge, system developers will be able to design information systems that will better mitigate the organizations business risks. Additionally, increased awareness of risk factors will provide company managers and system developers with greater opportunities to devise and implement appropriate and adequate controls during the development process, which should help minimize the cost of implementing controls (Lainhart 2001).

 

3    While systems risk analysis is clearly of interest to both auditors and systems professionals, we are unaware of research providing descriptive evidence on the nature of risks commonly present in business systems, and the implications of such risks for audit testing. This study addresses this research gap by examining two specific areas of information systems risk: management information quality and EDP security. 1  specific risk areas encompass the These physical and electronic integrity of client data systems, and the appropriateness of the information contained in those systems, respectively. Within each risk area, we document the frequency of specific system and client characteristics that auditors identify and consider when planning for a sample of actual engagements. Further, we assess the association of specific types of risk factors with auditors risk assessments, and with decisions to plan various types of audit tests. To conduct the study, we asked professional auditors to identify, for one of their actual clients, specific client conditions and issues associated with system security and information quality. The auditors also assessed the level of risk within each area, and noted the tests they would perform as part of the overall audit plan. Our results show that for a very high proportion of clients, auditors identify at least one risk factor in the EDP security and management information areas, even though this group of clients is assessed as being of relatively low overall risk. The most frequent types of risk factors identified in the area of system security are related to management attitude, keeping the system current, and maintaining adequate documentation. In the management information area, management attitude factors are also frequently identified, along with management competence,                                                            1  The two aspects of system risk that we consider (management information quality and EDP security) were chosen from the risk identification and assessment instrument of a Big 5 auditing firm. We chose to focus on two specific and important areas of systems risk in order to collect detailed data from participants, without making the research

 

4    and the nature and accuracy of information considered. Regarding audit tests planned in the system security risk area, we find that system-specific types of risk factors are statistically associated with planning of both tests of controls and review/inquiry procedures. In the management information risk area, both company-level and system-specific risk factors are statistically associated with planning review/inquiry and designing substantive tests. The implications of these results for auditors and systems professionals are discussed in the papers concluding section. BACKGROUND AND HYPOTHESIS DEVELOPMENT Auditing standards have long required that when performing the engagement, auditors obtain a sufficient understanding of the clients internal control (e.g., SAS No. 55; AICPA 1988). Recently, the Auditing Standards Board has issued SAS No. 94 (AICPA 2001), which expands and clarifies the auditors responsibility to understand the role of IT in the clients business (and specifically its control environment), and how the clients use of IT affects the audit strategy. Particularly, the auditors understanding of internal control supports a key audit strategy decision that must be made in every engagement: to what extent to rely on the clients controls. If the clients controls cannot be trusted, or if it would be inefficient to determine whether they can be trusted, the auditor may bypass the clients control system and apply direct substantive tests to the account balances. 2  If the auditor determines through documentation and testing that the

                                                                                                                                                                                            task unduly onerous. The two specific risk areas were selected by a focus group of experienced auditors; see the Methods section for further details. 2  While U.S. auditing standards previously permitted auditors to audit around the clients controls, this option may not continue. As part of its audit risk project, the U.S. Auditing Standards Board is considering the required nature and extent of testing of controls for all engagements. However, as noted above, the new Sarbanes-Oxley Act requires that auditors of public companies evaluate systems effectiveness in order to report on managements assertions about its systems, prompting near-term consideration of this issue at the professional level.  

5    controls are well designed and operating effectively, they may be relied upon and the extent of direct substantive testing of the accounts correspondingly reduced. A crucial component of the reliance decision is the evaluation of risk that the clients internal controls will not perform as intended. This paper examines two areas of that risk evaluation, management information quality and EDP security. First, if the output of the clients systems does not provide suitable and accurate information to support managements decisions, then the system may not signal when problems occur. If so, the monitoring function of controls is not achieved, and the risk of control failure is increased. Second, to the extent that these controls are computerized, lack of EDP security could negate any value in the effective design of the controls. In effect, security breaches can allow deviation from established procedures programmed into the system. Thus, in both risk areas, audit effectiveness will be increased if related risk factors are identified, and the level of risk assessed and tested, as part of the overall evaluation of the clients internal controls. Despite the importance of this task, prior research provides no direct evidence about the relationship of specific client characteristics to risk assessments and testing decisions in either area. In the following section, we review related studies and develop our hypotheses.

Research Hypotheses Our first hypothesis relates to the relationship between risk factors and risk assessments within each risk area. Auditing standards note that auditors should respond to engagement risks by increasing their risk assessments and altering the nature, timing, and extent of audit procedures (e.g., SAS No. 47, AICPA 1983; SAS No. 82, AICPA 1997). Prior research on the relationship of risk factors (specific client facts or issues) to risk assessments (summary

 

6    judgments of the level of risk) using behavioral designs generally finds that auditors overall

inherent/control risk assessments are related to differences in risk factors (e.g., Bedard and

Wright 1994; Davis 1996).Further, auditors fraud risk assessments are also associated with the

presence of risk factors that may indicate the presence of fraud (e.g., Zimbelman 1997). Few

archival studies of audit risk identify specific risk factors, but instead tend to focus on risk

assessments because these can be more easily compared across clients. An exception is Mock

and Turner (2002), who find that auditors fraud risk assessments are associated with the number

of fraud risk factors they identify in their clients. Based on prior research and professional

standards, we hypothesize that:  

H1. Within each systems risk area, the level of risk assessed is positively associated with the risk factors identified.   Our second set of hypotheses deals with the relationship between the two areas of

systems risk considered in this study, the security of EDP systems and the quality of information

they produce. While prior studies do not directly address this issue, it is likely that these types of

risk are correlated, due to common causal factors. For instance, poor system design can be due to

inadequate knowledge, insufficient effort/resources, or a combination of both. The level of

knowledge and effort applied to system development is likely to affect both the mechanisms in place for protecting data and the nature of the data that the system generates. In contrast, a well-

controlled business will provide adequate resources to secure and update systems in place, and

ensure that the information needs of management are being met. In support of this argument,

Haskins (1987) finds correlations in auditors perceptions of attributes related to client control

environments. Likewise, Waller (1993) finds similarity in auditors risk assessments across

assertions for a given account. These arguments lead to the following hypotheses:  

7 

  H2a. The number of risk factors identified in the two systems risk areas (EDP and Management Information) is positively associated.     H2b. The level of risk assessed in the two systems risk areas (EDP and Management Information) is positively associated.   The third research hypothesis considers the role of system risk factors in planning audit tests. As noted previously, auditing standards indicate that auditors should adjust the audit plan to reflect client risk factors. Regarding audit test planning decisions, prior research using behavioral methods often finds evidence of risk responsiveness in audit planning, consistent with auditing standards, (e.g., Libby et al. 1985; Houston et al. 1999; Asare and Wright 2002). However, results of archival studies on auditors risk responsiveness tend to vary, with some detecting little relationship between risk and audit planning decisions (e.g., Bedard 1989; Mock and Wright 1993, 1999) and others finding a relationship (e.g., Davis et al. 1993; Hackenbrack and Knechel 1997; Johnstone and Bedard 2001; Mock and Turner 2002). While the reasons for the mixed findings are not clear, there is sufficient literature supporting an effect of risk on audit planning to support a directional expectation for an effect of risk on audit planning. However, this relationship has not been tested in the systems context specifically, motivating our extension of the literature. We propose that:   H3. Within each systems risk area, identification of risk factors will results in increases in audit tests planned.   This research hypothesis anticipates a positive correlation of risk factors and audit tests. Detection of such a relationship is important, but it is even more informative to address which types of risk factors are associated with particular types of audit tests. In other words, how do auditors address particular types of risk in the audit plan? To address this issue, we provide

 

8    supplemental analysis relating the nature of audit tests to categories of risk factors within each risk area.  

METHODS Participant selection and characteristics Data for this study were collected from 46 auditors serving on engagement teams for 23 clients of two Big 5 accounting firms, in the presence of one of the authors. 3  and Selection scheduling of participants were accomplished with the assistance of a contact person at each firm, who was only aware that the study concerned audit planning. Participants responded to a questionnaire (described in the following section) about characteristics of one of their actual clients, which was selected in advance of the research session. Selection criteria were guided by the need for participants to have well-developed knowledge of client conditions. These criteria included the following: each participant must have completed at least one planning cycle for the client and be at least a senior on the current audit. Participants include 19 seniors or supervising seniors on the identified audit, 18 managers, seven senior managers and two partners. Their average length of audit experience is 6.9 years. Research task, procedures and variables The research task was developed with the assistance of a focus group of partners and managers from a participating firm. The focus group identified specific statements from the                                                            3 paper further analyzes data collected for another study, which studies the design of decision aids for audit riskThis identification, risk assessment and audit test planning. The prior study considers four specific risk areas, including the two areas of systems risk considered here. The primary goal of that study is to compare effects of decision aid design (i.e., a positive or negative decision aid orientation) on risk identification. Thus, we collected data from two engagement team members for each of 23 clients, who developed their responses independently. Because the goal of the current study is to present a detailed analysis of the type of systems risk factors identified, and their relationship to audit test planning, we combine data across orientations for most of the analysis. A dichotomous variable for decision orientation is not significant in the models presented in this paper. However, the paired design  

9    firms decision aid for risk identification and assessment. In ranking risk areas on appropriateness for the study, the focus groups considered such factors as the importance of the risk area in audit planning, its application to a broad range of clients, and its potential for differentiating more from less risky clients. Among these issues are the two systems risk areas considered in this study: (1) whether top management sufficiently oversees and addresses the risks related to data security and EDP system security for critical information systems; and (2) whether there are weaknesses in the relevance, completeness, timeliness and reliability of management information used by the company to monitor enterprise activity. These specific risk areas encompass the physical and electronic integrity of client data systems, and the appropriateness of the information contained in those systems, respectively. Within each area, participants first noted a summary assessment of risk on a seven-point Likert scale. Next, they were asked to list all specific risk conditions or issues involved with this client that you can recall, which you believe should be considered during the development of the audit program or are worthy of note by engagement personnel as the audit proceeds. The number of risk factors identified was determined from the client issues listed. Each issue was independently coded into the following types:Negative(a fact about the client that would lead to an increase in risk),Positive (a fact about the client that would lead to a reduction of risk), Neutral(a fact about the client that would not affect the level of risk). 4 The negative issues are termedSystems Risk Factors Identified. The risk factors are further divided into types (i.e., external factors, company factors and system-specific factors) in order to categorize the nature of                                                                                                                                                                                             allows additional insight, and so we also present supplemental analysis of differences in responses between pair members.