SAS 39 (AICPA 1981), Audit sampling," provides overall guidance on statistical and nonstatistical audit sampling, and requires auditors to take certain planning considerations into account when determining sample sizes. Subsequently, an Audit and Accounting Guide (AICPA 1983) was issued, also entitled Audit Sampling." It provides detailed guidance on how to apply SAS 39. A major portion of the Guide addresses nonstatistical sampling. In particular, the Guide provides a quasi- statistical formula (and tables) for determining nonstatistical sample sizes.

We present the results of a study that examines the effect of the nonstatistical sampling formula and table included by the Guide on auditor's sample size decisions and the variability, of those decisions. The study involves sample size decisions made by participants in three senior auditor training programs of two large accounting firms. The findings of our work are important for two reasons. First, based on a recent suryey by the Audit Sampling Implementation Task Force of the AICPA, the Guide is used for audit sampling applications by 28% of the 15 largest audit firms and by 68% of other large firms. Second, the AICPA has undertaken a project to revise the audit sampling guide.

Background

The Guide explicitly recognizes that in many audit contexts, a statistical approach to audit sampling may not be cost-justified. However, even where a nonstatistical approach is elected, the Guide states that the auditor is still responsible for selecting a sample size sufficient to reflect the risk of incorrect acceptance. For nonstatistical applications, the Guide provides a quasistatistical formula with the objective of illustrating "a method of assisting an auditor in gaining an understanding of the relative size of samples of substantive tests of details" (AICPA 1983, p. 58).

The Guide (pp. 59-64) suggests the use of the following formula:

Population'sBookVal.

SampleSize=------------------

-TolerableError

XAssuranceFactor

In this equation, tolerable error relates to the auditor's preliminary estimate of materiality. The Guide provides a table of six values for assurance factors based on combinations of three levels of desired audit assurance (substantial, moderate, or little) and two levels of error expectation ("little or no" or "some" errors expected). We refer to this formula as a decision aid because it is intended to assist the auditor's judgment

The AICPA formula is based on the statistical theory underlying probability proportional to recorded size (PPS) sampling (see Statistical Auditing, Roberts 1978, pp. 116-125; Audit Guide, pp. 74- 76). Because the probability of selection is proportional to recorded value, PPS is, in effect, a stratified selection of physical units. The choices of substantial, moderate, or little desired audit assurance correspond to PPS risks of incorrect acceptance of 5%, 10%, and 22.3%, respectively. For error expectation, the decision aid column for "little or no error" corresponds to a PPS critical error rate of zero, while the assurance factors for "some" error correspond to PPS critical error values between one and two sampled items for which errors are expected.

There is evidence in many fields of a cognitive bias in which individuals recognize that larger samples provide greater assurance, but understate the extent to which statistical power diminishes as sample sizes are reduced. in other words, people tend to underestimate the high sampling risk associated with small sample sizes. It is reasonable to expect that in the planning of nonstatistical audit samples, this bias might result in a preference for sample sizes too small to provide the desired audit assurance. To the extent that a size formula or decision aid might remove such a bias by forcing the auditor to concentrate on audit assurance and tolerable error rather than directly on the sample size, the use of such a decision aid would be expected to result in increased sample sizes. Thus, we tested to see if sample sizes recommended by auditors who used the formula were larger than intuitive sample sizes chosen by auditors who did not use any explicit sample size formula.

There is a second potential problem. Auditors can circumvent the AICPA decision aid by working backwards, starting with a desired sample size and forcing the requisite decision aid parameters in an attempt to justify a sample size that is fundamentally based only on intuition.

Such a strategy might be used by an auditor who is dissatisfied with the size of the sample generated by the formula and, therefore manipulates parameter values as necessary to yield a sample size that is more intuitively reasonable. We also tested this issue by asking some of the auditors in our study to provide only the input parameters of the AICPA formula (tolerable error, desired audit assurance, and error expectation) without actually computing the sample size. We later computed the implied sample sizes ourselves from the formula parameters provided by these auditors.

The rationale behind this approach is that if an individual is asked only for the input parameters of the formula, that individual will not have the opportunity to work backwards. That is, if auditors work backwards because they feel that the decision aid's output is unreasonable, one wax, to remove this incentive (or at least make it far less likely) is to ask auditors to supply only the inputs to the decision aid without calculating the output (sample size).

A final issue we investigated was whether the use of the formula affected the variability of the auditors' sample size decisions. Beyond systematic changes in the magnitude of judgments, a decision aid may affect the variability of judgments.

Method

Subjects. Two large auditing firms participated in the study. Auditor subjects from each firm were randomly assigned to one of the six groups discussed below. The instruments were administered at three senior-level training sessions. A total of 176 auditors with an average of 3.1 years of experience participated in the study. Twenty-four unusable subject responses were deleted, leaving 152 responses for our analyses.

Case Materials and experimental Conditions. The basic case used in all experimental groups included background information and a set of financial statements for a manufacturer of various parts for small consumer appliances. The auditors were instructed to direct their attention to the supplies inventory account, and were told that "the inventory of supplies is considered material to the financial statements taken as a whole, but is not so critical as to warrant formal statistical analysis." They were then told that the audit manager wanted to apply nonstatistical sampling to die subsidiary records. A list of proposed audit procedures was provided.

The auditors were also provided with information on the internal controls, expectation of errors, and other substantive tests for the supplies inventory. There were two internal control (and related error expectation) groups. The case information for the weak internal control group cited a number of control weaknesses, discussed prior-year weakness recommendations not completely corrected, and implied an expectation of error. The strong internal control group case cited implementation of new controls to correct the cause of prior-year errors. It was further stipulated that tests of controls indicated that the new controls were operating as described. The nature of other substantive tests was held constant in all experimental groups.

The basic task for the auditors was the determination of a nonstatistical sample size for the supplies inventory. The three experimental groups for the sample size selection task were: 1) an intuitive judgment group that provided a sample size without the decision aid (intuitive group); 2) a decision aid group that calculated sample size using the formula (decision aid group); and 3) a group that provided only the parameters for the formula with the researchers calculating the sample size afterwards (parameters only group). Descriptions of the steps of the decision aid and definitions of parameters were taken virtually verbatim from the Guide (pp. 59-60).

The combination of two internal control groups (weak and strong) and three experimental judgment conditions (intuitive, formula, parameters) resulted in a total of six groups. Each participating auditor was assigned to only one of these groups.

Results and Discussion

Table 1 provides descriptive statistics on sample size decisions for each of the experimental groups.

Formula Effects on Sample Size. The first issue addressed in this study suggests that the sample sizes of the intuitive group would be smaller than those generated by the decision aid group because the decision aid would force auditors to focus on the audit assurance that is implicit in the decision aid rather than directly on the sample size. The second issue is that some auditors might "back into" their desired sample size by manipulating the parameters of the formula. This would suggest that the sample sizes generated by the parameters only group would be larger than those of the other two groups.

The results showed that the auditors who used the AICPA decision aid provided significantly larger sample sizes than those who did not use the decision aid. Second, the sample sizes that were computed by us for the auditors who provided only the parameters to the formula were significantly larger than the sample sizes of auditors who used the AICPA decision aid and made their own calculations.

These findings suggest that the AICPA nonstatistical sample size formula has the effect of significantly increasing sample sizes above what auditors might otherwise consider to be intuitively appropriate. This conclusion holds whether underlying internal control is strong or weak. Evidence also supports what we refer to as a "working backwards" effect. That is, not only did the parameter judgments of the decision aid group affect their sample sizes; but the reverse was also true: desired sample sizes affected the choice of decision aid parameters.

Decision Aid Effects on the Variability of Sample Sizes. While the previous issues examine the effect of the Guide's sample size decision aid on the magnitude of sample size decisions, the final issue focuses on the variability of those decisions between auditors.

The results indicate that the sample size responses of the intuitive group were significantly less variable than the responses of auditors who used the AICPA decision aid formula. Thus, the Guide's formula increased the variability of sample size. This effect is the opposite of that predicted by intuition and other researchers. By way of explanation, it may be that the Guide's decision aid is not as "automated" as it might appear. That is, the nonstatistical sample size formula of the Guide simply replaces one intuitive judgment (sample size) with three other intuitive judgments (tolerable error, degree of desired assurance, and error expectation).

Much of the variability of the sample size distributions for the decision aid and parameters only groups can be attributed to the tolerable error parameter. This, in turn, suggests the importance of specifying financial statement materiality, If auditors differ as to the perceived magnitude of allowable error, then a sample size decision aid that is sensitive to materiality judgments will not generate consistency of responses.

Concluding Comments

This study examined the effects of a nonstatistical sample size decision aid on the sample size judgments of auditors. The decision aid led to systematically larger sample sizes than those elicited from unaided intuitive judgment. Sample sizes were even larger when subjects were asked only for the decision aid parameter values, suggesting that at least some subjects worked backwards in an attempt to circumvent the decision aid. Our tests indicated that the use of the decision aid resulted in more variance in auditor sample size judgments. Neither the magnitude nor the variability effects of the decision aid were sensitive to the level of internal control. The tolerable error parameter of the decision aid was responsible for much of the observed variability and for the difference between the sample sizes of the decision aid and parameters only groups, which we attributed to a working backwards phenomenon.

For more detailed information on the study the reader is referred to our paper in The Accounting Review in January 1990.