The Uniform CPA Examination is the only national professional licensure exam that is so disclosed. (1) During June 1991, The AICPA Board of Examiners voted to end this policy of disclosing questions and answers effective with the May 1996 exam. (2)

There are several reasons for adopting a nondisclosed process. Perhaps the most important reason is the potential for improvement in the quality of the CPA exam. The validity and reliability of the exam are directly dependent on the psychometric quality of the test items (i.e., questions and problems). Psychometric quality is the scientific way of saying the ability of an exam to achieve its stated objective of determining who should pass and who should not. The CPA exam leads to a pass/fail test score interpretation.

The three features of a multiple choice question that determine its psychometric quality are 1) its difficulty, demonstrated by the percentage of examinees who correctly answer the question; 2) its discrimination, demonstrated by the degree to which an item's answer distinguishes between examinees who know the most about the relevant topic and examinees who know the least; and 3) the functioning of wrong answer choices or distractors. The quality of an essay question depends on a number of factors: the nature of the problem selection, what is expected in the answer, the procedures that are followed in scoring answers, and how well the question discriminates between examinees who have the most knowledge about the topic and those who know the least.

Tested Items Promote Exam

Quality

Each item included on an exam should be reviewed for difficulty and discrimination. Disclosed exams are released to examinees and the public after each exam thereby forcing the test developer to use new and untested items in subsequent exams. Many of these items will not perform as intended. In a nondisclosed exam, many of the test items are pretested, and therefore tend to perform as intended. Anyone who has taken the SAT or ACT exams has most likely been a party to pretesting. There are from time to time questions on these exams that are being pretested. They are included on the exam but the answers are not counted in the grade for that particular exam. The psychometrician evaluates the answers to see if the questions accomplish the intended objectives of the test.

The Uniform CPA Examination deals with knowledge and skills necessary to enter the profession and serve the public appropriately. The AICPA has conducted a practice analysis to identify a content domain, i.e., knowledge and skills that are considered essential to professional practice. An examinee's test score on the CPA exam signifies the candidate's level of achievement relevant to this content domain. Comparing an examinee's test score to a standard pass/fail decision point leads to a conclusion that a person does or does not have the knowledge and skills necessary to protect the public interest and to enter the profession. Each test item must be selected according to test specifications to ensure valid test score interpretations.

With a nondisclosed exam, each item can be constructed to meet test specifications. Pretested items are more likely to perform according to expectations, satisfy test specifications, and validate the exam. If a test item is not pretested and fails to perform in a content area test, it will not satisfy content specifications of the test and will negatively affect content validity.

Reliability is based on a theory of measurement error and refers to the consistency or accuracy of the measuring process. Test items that fail to discriminate between competent and incompetent candidates tend to lower reliability. Lower reliability tends to increase the potential for misclassification of examinees whose scores fall at or near the passing score.

It is essential that a testing process be designed to increase reliability in order to improve the precision of the test at the critical passing score in order to minimize the potential of misclassifying competent candidates as incompetent and an incompetent candidate as competent. A way to increase reliability is to ensure that each test item is contributing useful information. Test items with faulty psychometric characteristics contribute to lowering reliability and to increasing the amount of measurement error in exam scores, which in turn increases the potential for misclassifying candidates whose true scores fall near the critical pass/fail score.

Adoption of a nondisclosed exam process would allow the CPA exam to be comprised of pretested items that have known psychometric qualities which will not lower reliability.

In addition to improvement of validity and reliability of the exam, the CPA Examination Review Board (ERB) of the National Association of State Boards of Accountancy in their position statement "Why the Uniform CPA Examination Should Be Secure," indicates an essential requirement of a licensure exam is that the exam be comparable from one administration of the exam to the next. ERB points out that the statistical process of "equating" ensures that test content difficulty is comparable from administration to administration; equating disclosed exams is not practicable since there is no way to ensure use of a significant set of common items from exam to exam because there are no pretested items. (3)

Pollution Control

Another matter of concern related to use of a disclosed exam is the matter of test score pollution. ERB defines test score pollution as practices that boost or diminish test scores without a corresponding change in a candidate's mastery of the content domain. ERB points out that one way to pollute test scores is to teach to the test and that released items from previously administered exams that serve as the basis of large scale coaching clinics can cause test score pollution. (4) It is not in the best interest of the licensure process to have any factor that biases interpretation of the exam scores. A nondisclosed exam would alleviate this problem. The continuation of a disclosed exam will perpetuate and increase test score pollution as the pool of previously used test items increases and specific test item coaching becomes more pervasive.

While many educators lament the effect of the Uniform CPA Examination on the accountancy curriculum, the exam may be more responsible than many are willing to admit in preserving the rigor of the accounting curriculum absent effective program approval and accreditation as components of the licensure process, while the curricula of many other academic disciplines were becoming soft. There is no reason why this beneficial effect cannot be perpetuated. A periodic release of exam items will fulfill educational needs. Kane points out that "where feasible, the release of sample copies of the exam would serve a useful function in informing discussion and debate about licensure. Complete disclosure of all exams is probably not justified in most cases...because the additional benefits of complete disclosure compared to partial disclosure would be minimal." (5)

Adoption of a nondisclosed exam process will allow test items to be pretested and the quality of the exam to be improved. Statistical equating methods can be used with a nondisclosed exam that will ensure comparability of exams. In addition, ERB points out in their position paper that the nondisclosed exam is more cost effective than a disclosed one in which new test items must be constantly created. "In a (nondisclosed) testing program, items are initially written, pretested, evaluated, and then assigned to future exams on a systematic basis. This process ensures that only the finest, best performing items appear in a test. Over a period of years, the pool of test items increases. The cost of maintaining or increasing the pool is relatively small as is the cost of constructing new test forms for each test date. Therefore, the (nondisclosed) exam is more cost effective." (6) Omega

Charles W. Taylor, PhD, CPA, of the University of Mississippi, is past Chairman of the NASBA CPA Examination Review Board.

REFERENCES

1 Exposure Draft, Proposed Changes in the Uniform CPA Examination, Board of Examiners, American Institute of Certified Public Accountants, New York, 1987.

2 Position Paper, A Nondisclosed Uniform CPA Examination, Board of Examiners, Examinations Division, AICPA, New York, September 19, 1991.

3 Report of the CPA Examination Review Board 1989-1990, NASBA, New York, August, 1990.

4 Ibid., p. 10.

5 Michael T. Kane, "The Future of Testing for Licensure and Certification Examinations," in Blakes, B.S. and Whitt, J.C. (Eds.) The Future of Testing, Hillsdale, NJ: Lawrence Erlbaum Associates, 1986.

6 Report of the CPA Examination Review Board 1989-1990, NASBA, New York, August, 1990, p. 11.