Approach to Testing Effectiveness

For the study, we used the reported quarterly financial statement data of five single-industry companies to test the effectiveness of analytical procedures. Our approach was to include a material misstatement in a quarter's reported results. The measures of materiality considered were: 10% of current income, 10% of average income over a three-year period, .5% of sales, and an approximation of KPMG Peat Marwick's "audit gauge." (See K. Pany and S. Wheeler, "A Comparison of Various Materiality Rules of Thumb," The CPA Journal, June 1989, pp. 62-63.) Then, we used an investigation rule (e.g., investigate when the account balance changes by more than 5%) and determined whether it indicated to investigate or not to investigate. When the rule suggested investigate, we considered it to have operated effectively, since we knew the account was materially misstated. Alternatively, when do not investigate was suggested, the rule was considered to be ineffective in detection of the material misstatement.

After completing the above, we deleted the misstatement from the account and again performed the analysis. When the investigation rule suggested investigate, the investigation rule was considered to have failed since no misstatement was included in the balance. Alternatively, when the rule indicated do not investigate, it was considered to have operated effectively.

The above approach was used in a variety of contexts for each of the five companies in the study. The types of misstatements, accounts, and ratios tested are presented in Figure 1. We considered material misstatements first from the perspective of the quarterly results and then from the perspective of annual results.

Results Based on Quarterly Materiality

After applying the investigation rules we calculated the percentage of the time no investigation was indicated when the account was misstated. Similarly, error rates were generated for the situations in which there was no material misstatement. Figure 2 presents summarized results for three separate investigation rules. The first two are "simple change rules" that call for investigation of changes of more than 5% and 10%, respectively, from the amount that had been expected. The "statistical" method uses the standard deviation of each account or ratio to define unusual variations from predicted amounts.

To illustrate the interpretation of Figure 2, consider the 5% rule. The table indicates that 61% of the time the procedure failed to identify the existence of a material misstatement.

Figure 3 presents results for the situation in which we deleted the material misstatement. The 32% indicates that when no misstatements were present, the 5% change rule still suggested the presence of a material misstatement that percent of the time.

Results Are Not Encouraging

The quarterly results, taken in isolation, are not encouraging for these analytical procedures. Sixty-one percent of the time the 5% change rule did not trigger investigation when in fact a material misstatement existed; and 32% of the time it said to investigate when no misstatement existed. Most auditors would consider the potential costs of such non- discovery to exceed the costs of the unnecessary investigations indicated in Figure 3. Therefore, analyzing Figure 2, it appears that most would view the statistical rule as having performed more effectively than the percentage change rules. For example, as seen in Figure 2, the average error rate using even the best investigation rule was 29%. On the other hand, the statistical rule said to investigate 64% of the time when the misstatement had been removed. Error rates of the magnitude shown in Figure 2 would normally be considered high for substantive audit tests. However, if analytical procedures are combined with other audit tests, these error rates might well be deemed acceptable, given the efficiency of analytical procedures and given audit consideration of inherent and control risk factors. That is, given specific assessed levels of inherent and control risk, the combination of all substantive audit tests performed might adequately restrict detection risk in a cost-effective manner. But, in most contexts, relying entirely upon such analytical procedures would seem risky.

Figure 4 further categorizes the average error rates (using the statistical investigation rule) when a misstatement has been included by ratios, accounts, balance sheet accounts only, and income statement accounts only. Comparing these groupings showed that the average for ratios (30%) was significantly higher than the average for accounts (26%).

The performance of one misstatement type and ratio combination deserves particular attention. Specifically, the interest expense to interest-bearing debt ratio failed to detect only one misstatement in 52 sample company quarters examined (3% rate). No other misstatement type/ analytical procedure combination distinguished itself similarly across all sample companies. The second-best analytical procedure performance was in the depreciation miscalculation misstatement analysis. For both interest and depreciation, a typical substantive test is recomputation. Therefore, the evidence suggests that analytical procedures are most effective in situations where the primary substantive procedure would be recomputation.

Results Based on "Annualized" Approaches

To this point we have presented the likelihood of misstatement detection when only one quarter of a year has been misstated by exactly a "quarterly" material amount. In an audit of financial statements, more than one quarter may be materially misstated, and misstatements in excess of a "quarterly" material amount may occur in one quarter. Therefore, the results are conservative from an "annualized" standpoint since, in practice, an investigation in one quarter may lead to further analysis and detection of misstatements in other quarters. Also, with quarterly material misstatements, an auditor would generally need to detect and adjust only one misstatement to render an account materially accurate on an "annualized" basis. Furthermore, if a material "annual" misstatement is concentrated in one quarter, analytical procedure effectiveness may be expected to improve.

Whether an annual material misstatement is spread over four quarters or concentrated in one quarter will vary. Accordingly, there is no clear answer concerning whether four "quarterly" material amounts or an annual material amount should be included in the analysis. To address this issue, we supplemented the analysis in two ways. First, we applied four possible annualized investigation rules to quarterly results each misstated by a quarterly material amount. Second, we included "annual" material misstatements in the fourth quarter of the two prediction years.

Our four "annualized" quarterly rules required at least one, two, three, or four quarterly investigate indications, respectively, before the year-end balance would be tested. We found that a rule which investigates when at least one quarter indicated investigate detects material misstatements 99% of the time across all analytical procedures. However, when there was no misstatement, such a detection rule led to investigations 96% of the time. In short, such a rule almost always tells the auditor to investigate. The result is a procedure that is not very useful. Our findings are consistent with the argument that analytical procedures may be good at spotting the presence of misstatements, but do not reliably indicate the absence of misstatements. In other words, analytical procedures are more effective in audit planning than when used as substantive tests. When more than one quarter was required to lead to a decision to investigate, the percentage of incorrect do not investigate rules increased to 25%, 45% and 78%, respectively, for rules suggesting two, three, and four quarters.

Under our second approach, an "annual material misstatement" was included in fourth quarter balances for the two prediction years to simulate material year-end cutoff errors. The incorrect do not investigate rate across all analytical procedures tested was 8%. When the misstatement was removed the incorrect investigate rate was 72%. Consistent with prior studies, these results indicate that large, isolated misstatements are much more likely to be detected using analytical procedures.

Implications for Using Analytical Procedures

The study calls to question the likely effectiveness of the mechanical use of simple analytical procedures. In the test of misstatement of quarterly results, the analytical procedures generally were not that effective. However, when the quarterly results were interpreted from an "annualized" perspective, and when "annual" material misstatements were included in year-end quarterly balances, the results were much more promising.

It seems (and research indicates) that SAS 56 has led to increased use of analytical procedures in all areas of the audit, including substantive testing. Also, surveys indicate that most auditors tend to use relatively simple methods such as comparisons with prior years' balances. The difficulty with expecting simple change rules to detect a material misstatement is that typical materiality measures result in amounts much lower than the simple 5% and 10% change methods can detect. In fact, we found that on average, the 5% and 10% change rules would have required a misstatement equal to four to ten times or more the computed materialitv measures before the error was noticed. Consequently, if auditors practice analytical procedures using simple expectation models and simple investigation rules (e.g., 5% or 10% change methods), it may be reasonable to expect that only very large misstatements will be detected.

Statistical investigation rules that use more information about prior data potentially may be helpful. Simple 5% or 10% change methods use only the most recent prior balance as an expectation for the current balance. However, when using several years of interim data, certain within-year or between-years trends or patterns may be present and may result in more effective analytical procedures.

While our results do indicate that simple change rules often may be ineffective, they should not be interpreted as suggesting that only statistical models are likely to be effective. What may be necessary is that auditors give detailed consideration of their knowledge of the client's business. For example, although not directly tested in this study, rather than comparing a simple change in sales, given strong internal control, a more effective procedure might be to compare unit records of goods shipped with average prices during the year to determine whether sales appear reasonable. Similarly, a measure of payroll expense might be calculated using average wage per a union contract multiplied times a measure of the average union employees during the year. Another useful factor is physical capacity constraints. By comparing recorded inventory quantities to warehouse capacity limits, gross overstatements may be detected. Similarly, sales quantities can be compared to production capacity limits for the given time period.

Be Very Careful

It is important to realize that the approaches tested in this study did not take into consideration that auditors may have a wealth of information to use in addition to the commonly used simple change rules. For example, the auditor who is aware of certain industry conditions may have a much more precise estimate of expected account and ratio values. Therefore, if other, non-financial information (e.g., production data) had been available, the accuracies of the methods tested might improve. Also, any additional company-specific information which might be obtained through inquiry by an experienced auditor could not be captured. Thus, the effectiveness of analytical procedures in locating misstatements may be underestimated by considering the results of this study in isolation. Also, since it is likely that these procedures would be coupled with other detail tests or other analytical procedures, the noted error rates are not necessarily correspondent with achieved detection risk. Despite these caveat,s, it is clear, at least for the companies tested, that the auditor who uses simple percentage-change methods should place extremely limited reliance on their results.

More sophisticated analytical procedure techniques may enhance audit effectiveness, especially when employed in conjunction with some minimum level of traditional substantive procedures. Outcomes in which the statistical methods dominated the results were commonplace throughout our study. However, even then, caution should be exercised when determining the degree of reliance to be placed on analytical procedures when used in isolation.

Overall, the results imply that the investigation thresholds using simple percentage change rules are normally much larger than what would be considered a material misstatement to the financial statements. But, even the more sophisticated statistical methods we employed missed numerous misstatements. Therefore, this limited effectiveness would seem to imply the need for auditors to carefully design analytical procedures to incorporate their overall knowledge of the client. Certainly, only limited reliance should be placed upon the simple change rules included in this study.

By Kurt Pany, Arthur Andersen/Don Dupont Professor, School of Accountancy, Arizona State University; and Steve Wheeler, Assistant Professor, Florida State University