ASNT Certification: Psychometrics

All ASNT examinations are developed using psychometric principles. Psychometrics is the field of study concerned primarily with the construction and validation of questionnaires, tests, and personality assessments, with guidelines found in the Standards for Educational and Psychological Testing, developed jointly by the American Educational Research Association (AERA), the American Psychological Association (APA), and the National Council on Measurement in Education (NCME). Dr. Larry A. Early, PhD, a psychometrician and former ANSI ISO 17024 auditor, is ASNT's contract psychometrician.

Development Process 

ASNT's NDT Level III examinations are developed and maintained by the Certification Management Council (CMC), a volunteer group of NDT subject-matter experts (SMEs) from multiple industries holding current ASNT NDT Level III certificates in one or more NDT test methods. They develop the examinations using psychometric principles to ensure that they are fair, valid, reliable, defensible and compliant with accreditation requirements (ANSI 17024).

All ASNT examination questions are developed on the subject matters listed in the Topical Outlines (or "Body of Knowledge") found in the ANSI/ASNT American National Standard CP-105, ASNT Standard Topical Outlines for Qualification of Nondestructive Testing Personnel. Each question must be attributed back to a specific line item within the Topical Outline for the applicable test method and a reference showing where that piece of knowledge can be found must be identified. All reference materials must be within the public domain (i.e. available to the public).

As part of the development process, the CMC members create an examination specification that determines how many questions on an exam will come from each section of the Body of Knowledge. This specification is used every time a new examination form is created and ensures that each form tests the candidates on the same percentage of questions per subject category.

Once a question is developed, it is evaluated by a committee of CMC members certified in that test method to ensure that the question content is valid, i.e. that it will adequately test the candidates' comprehension of a subject determined to be applicable to the knowledge expected of a Level III in that test method. Each new question is pretested by placing it on a current exam form to gather statistical performance data (and candidate comments/critiques) and it is not scored for or against a candidate. After approximately 75 exposures, each new question is re-evaluated by the CMC to verify that they meet the acceptable statistical criteria of actual performance. 

Only questions with difficulty values (the percentage of correct answers to that question) from 0.3 (a relatively difficult question) to 0.95 (an easy question) are accepted and deemed validated. 0.3 is used as a lower limit because with four possible answers, the laws of probability give candidates a 25% (0.25) chance of answering a question by guessing.

Passing Score Development

In order to set an objective, criterion based passing score (a cut score) the Angoff procedure is followed. The Angoff procedure is a generally accepted psychometric procedure widely used by many certification programs to set a defensible passing score. The Angoff procedure is as follows: each CMC member independently estimates a percentage of minimally competent inspectors who would correctly answer a validated test question. The estimated percentages are then averaged and that average becomes the "Angoff value" for that question and it is entered into the appropriate examination databank with that question. When a new examination form is generated from a databank, the Angoff values of the validated questions selected for use on that form are averaged and that number becomes the passing score for that examination form. The resulting passing score for the exam form is typically in the 70 to 80 percent range. 

Because some of the questions are replaced on each new examination form, the average difficulty level of each examination form will be slightly different. However, since the examinations are based on the same examination specification, the passing score will always represent the minimum knowledge required. Most importantly, since the absolute knowledge standard is held constant, the ability of a candidate to pass an exam is not affected by any other candidate's performance (as would occur if a bell curve were used), and the passing score is not set against an arbitrarily set fixed percentage.

Use of Psychometrics

ASNT is not alone in the use of psychometrics in the development of examinations and cut scores. The American Petroleum Institute (API), the American Society for Quality (ASQ) and the National Council of Examiner for Engineering and Surveying (NCEES) all use a form of psychometrics for their examination and cut score development. The API exam processes can be found online at www.api.org and ASQ's exam processes can be found online in their Certification Handbook at www.asq.org

Accreditation

ASNT is accredited by the American National Standards Institute (ANSI) in accordance with the international standard ISO 17024, Conformity assessment - General requirements for bodies operating certification of persons. This standard is recognized worldwide and requires frequent auditing by the accreditation body to ensure that all of the requirements, including fairness, reliability and validity (psychometrics) are met.