Probability of Detection - An NDT Solution

If you have been having second thoughts about the challenges of a probability of detection study, this article will help you decide. The author has beautifully addressed the key questions: What is it? Why do it? and How can it be done? Ripi Singh Guest Editor

 

Introduction
The need to solve nondestructive testing (NDT) problems in the aging aircraft community appears to grow faster than the rate at which aircraft age. New NDT techniques and technologies are required to solve these problems. The new solutions must be transferred from the laboratory to the aircraft as quickly as possible. Unfortunately, many aircraft will be retired from service before the new NDT techniques can be implemented. This paper offers the logic of using the probability of detection (POD) demonstration process to accelerate the transference of NDT techniques developed in the research and development laboratory into the field. This enables the validated technology to be applied to the production environment in a timely fashion. While the technique can be used for any industry requiring new, cost effective and validated NDT procedures, this paper uses examples that target the aging aircraft industry.

The process of taking a new NDT technique from the research and development laboratory to final implementation on an aircraft is lengthy, expensive and difficult. This paper examines the many steps required to correctly accomplish that process, why attempts can fail and how conducting a POD demonstration gets the job done quickly while validating the entire procedure. The importance of utilizing a team concept to solve this type of problem and the various roles team members play in a successful and cost effective solution will be discussed.

---

The upside of conducting the POD demonstration is the rapid transfer of the technique to the production line. 

---

 

A comparison will be made between the approach using a POD demonstration and alternative approaches taken on the same task. Costs of accomplishing both procedures will be compared and analyzed. The risks of not developing a method to validate the NDT technique before applying it to an aircraft will be considered. Examples from recent POD studies will be discussed, including the time required to apply the techniques on the aircraft and the cost savings resulting from their implementation.

 

Defining the Problem
In all engineering, problems cannot be readily solved until they are completely defined and understood. Classic NDT problems are no different. One of the most repeated questions engineering management asks is: "Why do we spend so much for research and development while few of our efforts make it from the lab to production lines?" Many ideas end up working in the laboratory but are not successfully transferred to the production line. Engineers usually focus on the details of a problem, rather than the big picture. They need to fully understand the complete problem and the parameters to be considered before seeking a technical solution. NDT problems demand greater focus due to the limited number of technologies that are available when selecting a possible solution. It is easy to get lost in details without considering the total requirements of an application and the implementation of the engineering solution selected to solve the original problem.

When the engineering community usually thinks of POD experiments, they first envision a set of curves. These curves demonstrate that a certain size discontinuity can be detected with a certain probability at a given confidence level. This is the concept from the statistical side of the engineering discipline. While this is true and extremely important in validating the techniques selected, the process of properly conducting the POD demonstration is really a microcosm of the entire testing process. In engineering, the answer usually lies at the end of a body of work. In this case, the answer is located in the many steps involved in the body of work that must be performed to arrive at the final answer. It is possible to implement a nonvalidated NDT technique on an aircraft without a POD; it has been done for many years. However, if you cannot conduct a proper POD demonstration on a series of specimens, you cannot be expected to implement a validated application on an aircraft. More of this theory will be explained later in this paper.

 

Probability of Detection and Probability of False Alarm
To use a classic definition: probability of detection, as a function of the discontinuity size, is the fraction of discontinuities of a nominal size that are expected to be detected. This is normally expressed as a ratio of a probability of detecting a discontinuity with a confidence level 90/95, 90/90, 90/75 or 90/50, depending on the requirements of the application. The first number in the series denotes the probability that the anomaly will be detected, which is given as a percentage. The second number denotes the confidence level for detecting the anomaly. This information is usually represented as a graph. The probability of detection is plotted as a function of anomaly size for a fixed confidence level. Figure 1 represents some typical POD curves based on a 90 percent probability and a 95 percent confidence level.

Figure 1 - Typical probability of detection curves.

 

A POFA is the probability of a false alarm. If a discontinuity is called when there isn't one, a false alarm has occurred. In a perfect world, any detection scheme is planned to find the actual discontinuities without missing rejectable anomalies and not calling a discontinuity where none exists. In truth, most detection schemes are set to insure that a discontinuity is not missed, while keeping the false call rate at less than one percent of the calls being made.

 

Probability of Detection Demonstration
In the field of aging aircraft, the data used to set up the experiment is developed from two documents. The first is the Reliability Assessment at Airline Inspection Facilities, Volume 1: A Generic Protocol for Inspection Reliability Experiments. Airline experts developed this document after the Aloha Airlines accident in 1988. The Federal Aviation Administration (FAA) released it in March of 1992. It has been used as a guide in setting up POD demonstrations on aircraft structures since its inception. It was used for the POD demonstrations that are referenced and discussed in this paper. The second document is the US Air Force Aeronautical Systems Center's Military Handbook MIL-HDBK-1823, Nondestructive Evaluation System Reliability Assessment (1999). This document contains, among other things, the number of specimens, with and without discontinuities, required to determine a certain probability and confidence level for a testing technique.

This discussion will go through some generic steps that must be considered to set up and successfully conduct a POD demonstration. Each demonstration will contain special procedures dedicated to specific needs. The general list serves as a good checklist of what is included in the process. That list follows the process that must be considered to develop a full and complete testing procedure that will apply NDT technology to an aircraft. As in any assignment, specific goals must be met at the end of the task. Planning the goals for accomplishing the POD demonstration is similar to planning the implementation of the new NDT technique onto an aircraft. Planning those goals is described in the following steps.

Determine the type and size of anomalies to be detected. The structural engineers responsible for the design and/or maintenance of the aircraft's structural frame determine the basic requirements for this information. These include a range of sizes that the testing process must detect, as well as smaller indications that would not normally be detected by the NDT techniques selected for the demonstration. This gives the NDT engineers initial test goals, as well as measuring the true detection capabilities of the NDT techniques selected for the testing process. The probability of detecting the required discontinuity size with a confidence level (usually 90/95) must also be established as part of the initial requirement of the demonstration. This criteria is normally supplied by the aircraft's structural analysis team and has factors of safety built into its selection.

Using the amount and direction of stress as it applies to the component, together with other engineering data, to determine the orientation in which the anomalies can be expected to grow is an important part of the detection process. Making use of available engineering data is an important feature, often overlooked, in solving testing requirements, speeding up the testing process and reducing the amount of area to be tested. For example, asking stress engineers to determine which fasteners in a structure will undergo the highest stress levels has reduced the number of fasteners initially requiring testing from 200 to 8, 94 percent less than the original requirement. Asking the correct questions at the beginning of a task can greatly reduce false starts and eliminate bad assumptions. Making use of all the available engineering data at the beginning of a program can have a large impact on solving the problem more quickly with less effort and at lower cost.

The NDT engineer usually determines the solution of an NDT problem. Due to the pressure of a tight time schedule, the NDT engineer often selects the most familiar and not necessarily the best method to accomplish the task in the most efficient and cost effective manner. To solve NDT problems effectively, in this day of ever improving technology, NDT engineers must be provided the tools and the time to master their application. They can then be proactive in applying solutions to complex NDT problems, such as those facing the aging aircraft fleet.

By far, the most costly part of any POD demonstration is the collecting or manufacturing of specimens with and without discontinuities that represent the actual components to be tested. Keep in mind that these specimens must contain a distribution of minimum and maximum sized anomalies, as well as large areas without any anomalies. To make the POD valid, the number of indications available is usually in the range of 100 or more, with two to three times that number representing regions without discontinuities. While actual anomalies, as frequently found in the components to be tested, would be ideal and the best with which to work, often the cost of duplicating those anomalies is extremely high. A substitute method, which may differ slightly from reality, can be accomplished by using electrical discharge machined notches or other schemes for embedding artificially created discontinuities.

Also, it is important to completely test the hardware and its associated software to insure proper integrated operation. This usually is the task that NDT engineers enjoy and thus it receives a great deal of attention, as they enjoy working the equipment phase of a program. They often conduct this phase very well. A weakness is that they may not be equipped with the latest NDT equipment to solve the problem. It is therefore important that correct NDT techniques be made available to insure that the most efficient solutions be applied to the task at hand. Figure 2 depicts a typical laboratory setup for a POD being conducted at the University of Dayton Research Institute.

Figure 2 - Automatic couplant ejection system being tested at the University of Dayton Research Institute.

 

Each agency has its own method for keeping track of methods applied in each test. In the US Air Force, the procedures are called technical orders; in the commercial world they are referred to as the airworthiness directive. When more than one technique can accomplish the same task, later versions are usually called alternate means of compliance. The manufacturer and/or the FAA issue these documents, normally approved by the airframe manufacturer. It is vital that the approving authority, before conducting the POD, approves any procedures developed for it. If the agency does not approve the technique, the work could be invalid and not be allowed for use on the aircraft in question.

In order to duplicate actual field conditions, the POD must be conducted in environmental and physical conditions that are as close as possible to what the inspectors normally encounter on the aircraft. This includes location and access to the areas being tested, lighting conditions and temperature. A great deal of effort goes into briefing each inspector taking part in the demonstration. Each inspector must fully understand the selected protocols. Testers must be fully trained at their required level of NDT competence, especially on the technique to be used for the POD. Testers will not be graded on their performance as there is no pass or fail grade on a POD demonstration.

 

Calibration Standards
All NDT procedures require a standard to calibrate both equipment and sensors, insuring that the operator has the NDT technique set up to detect the range of indications required for the test. In most cases, actual anomalies cannot be used because of the lack of repeatability from anomaly to anomaly. The electrical discharge machined notches or slots or side drilled or flatbottom holes machined in the same or similar materials have been used for many years as
repeatable targets for calibration standards. In the case of fleet aircraft tests, one or two calibration standards are required for each test kit and additional standards are required for training. The need for uniform calibration standards remains important. With the advent of more POD demonstrations used to validate NDT techniques, the calibration standard becomes more significant. To eliminate variables introduced by using multiple calibration standards, testing the same standard increases the level of consistency during a POD demonstration. While this helps to produce uniform results for the POD, it does not check the differences between standards. Small differences do not always affect test results. However, as structural requirements of finding smaller indications increase, the calibration standard takes on a more important role. It is suggested that all of the calibration standards be tested using the same technique to insure their repeatable output, meeting the intended requirements.

 

Selecting and Training Qualified Operators
A limiting factor in bringing techniques from the laboratory to the production line has traditionally been the skill and experience of the personnel, especially in the use of a new technique. Many problems solved by laboratory technicians or engineers cannot be easily duplicated by other technicians in that same laboratory, let alone by production trained inspectors. Reasons vary, from the time it took to develop the technique, the experience of the operator, knowledge of the equipment and varied working conditions. These factors are usually very different in the lab than when compared to those factors experienced in the field. Another factor is that the sample on which the technique is being developed represents only a few of the actual number of indications of the size and orientation that are found in the real world. Solving this same problem using 50 or 100 anomalies, as would be done during a POD demonstration, is a more universal solution to the problem. It insures greater detection capability for a broader selection of inspectors. For this reason, a wide range of inspectors should be selected who vary in experience, age, skill, training time and time between training and certification, to insure a good cross section of operators. In addition, it should be noted that choosing production inspectors to participate in the POD program is an excellent method of introducing a new testing technique to the production team. This allows team members to actively participate in the solution of a problem, making the new NDT technique more acceptable. The production inspectors can become the in house champions of the new technique, helping to insure the success of the new testing program. Should results determine that the POD is less than satisfactory or too difficult to set up and/or operate, the inspectors become part of the solution. They contribute suggestions to correct the problem and improve the procedures and equipment. Either way, the POD demonstration will prove or disprove the technique before going into production, insuring that a fully verified technique is placed in service.

 

Conducting a Parametric Study and Probability of Detection Demonstration
A parametric study is conducted to better understand the method by which changes to major setups of procedures affect the final test results. This type of study continuously reruns the experiment over a known sample, making only slight changes to a dedicated number of parameters to determine the limits that a parameter can be altered while achieving acceptable results. Parameters that are usually changed include the alignment of the scanner to the component, expressed in ± 0.5 degree steps, and the overall gain to the amplifiers, usually expressed in ± 0.1 dB steps of gain. Depending on the NDT technique, different parameters may be included in the study. The objective is to determine the range of the parameters of the NDT technique that can be used to insure accurate acquisition of data, which allows the user to make appropriate and correct decisions.

After all the preparation, the time has arrived to conduct the actual POD. Samples are assembled in a representative method to simulate what will actually be found on the aircraft. The specimens are usually represented in the same size format, so they can be moved from location to location to keep the identity of the various anomalies unknown to the inspectors as they examine them from shift to shift and day to day. Before the POD experiment is run, each inspector has been trained on similar specimens to insure proficiency in the setup, operation and analysis of the system being tested. On the day of the POD experiment, each inspector is interviewed, briefed and given instructions regarding the experiment. Time limits, while not a major limiting factor, are explained. Testers will have sufficient time to complete the experiment. Each inspector is assigned a special code number, which is kept anonymous to the individuals making the final analysis. These data are then reassigned a new reference number, so that complete anonymity is maintained between the data and the individual taking the data. During the experiment, each operator is observed and a record made to support the POD demonstration. Any abnormality that could affect the data is observed and recorded; however, the inspector's work is not interrupted unless there is an equipment failure or the operator cannot continue for some reason.

When the inspector has made an analysis and turned in the records, the records are sealed and turned over to an independent authority to analyze the data. This is an extremely important feature of PODs. Personnel completely independent of the data acquisition phase of the experiment should analyze the data. This insures that individuals can make an unbiased determination and that the POD curves are true and accurate.

When the data are analyzed and the curves plotted, did the results of the experiment validate the NDT technique and meet the expectations of the experiment? If they did, prepare an implementation plan, put it into action and start enjoying the return on investment. If they didn't, review the entire procedure to understand just what went wrong. If the process is still valid, make the required changes and rerun the experiment. Since 95 percent of the work has already been done, making slight changes to the procedure will add very little to the overall cost and the program can achieve success. If during the analysis the selected NDT technique appears to be incorrect, or an additional technique must be added to detect all of the anomalies possible, then a major overhaul is still possible because the most expensive part of the POD is the specimens. They are still available for use with the new or replacement technique. Success is within reach. Make the required changes and rerun the experiment.

If, after all of the above, it's unclear as to what went wrong, the difficult decision must be made — cancel the program and find an alternative method to solve the problem. The fact to remember is: if you couldn't make it work for the POD, then it certainly would not have worked on the aircraft.

 

Cost Analysis of Probability of Detection Demonstrations
This section will discuss cost estimates for conducting six POD demonstrations since 1992. The estimates are shown in Table 1. The first one represents an FAA experiment conducted at nine commercial facilities to determine the effectiveness of using handheld eddy current to find first layer cracks emanating from lap joint fastener holes on Boeing 737 aircraft. This experiment was conducted in response to the Aloha Airline incident in 1988. It was the first major POD run on aircraft structures since the Air Force's "Have Cracks, Will Travel" program conducted in 1979. The second column shows the cost of conducting a POD demonstration for the C-141 wing splice joint second layer test. This program, started in 1996, was completed in 1998 and has been in production at Warner Robins ALC since June of 1999. The technique has been validated to find 1.8 mm (0.07 in.) cracks in the second layer of fastener boltholes in the splice joints. The estimated savings for the splice joint test, once the entire fleet has been tested, is $38 million. This new technique, which validates that a smaller anomaly could be found, increased the time between tests from four months to five years.

	FAA-737 Eddy Current	C-141 Second Layer SJ	C-141 First Layer SJ	C-130 Hat Section	C-141 Image Weep Holes	C-141 NN Weep Holes
Approximate Cost of POD	$1,500,000	$800,000	$250,000	$350,000	$250,000	$350,000

Table 1 Cost of conducting six different POD demonstrations

A second POD was run on the C-141 splice joint to determine if it was feasible to simultaneously test the first layer while the second layer was being tested. Basically, a second gate was added to the testing procedure and the first layer was tested at no additional cost or time. This technique eliminated a complete testing cycle for the entire fleet. The POD validated that 1 mm (0.04 in.) anomalies could be detected in the first layer at a 90/95 percent probability/confidence level.

The hat section on the C-130 center wing section presented a similar, yet different, problem than the C-141 splice joint. The C-141 splice joint presented a smooth surface to test, while the C-130 has thousands of buttonhead fasteners that extended normal to the surface as much as 4.8 mm (0.2 in.). To enable the testing process to be conducted over the fastener heads, a special automated couplant ejection system was developed. It slides over the protruding fastener heads while supplying an approximately 25 mm (1 in.) water column to two transducers. The surface is left completely dry and safe to walk upon immediately after the couplant ejection system has traversed over an area. Figure 3 depicts the system setup 8 m (25 ft) in the air on a C-130 center wing section. The level of safety provided by the system is demonstrated by the location of the photographer, whose shadow is visible in the picture.

Figure 3 - Couplant ejection system on wing setup.

 

The last two PODs were conducted for the C-141 external test of weep holes. An earlier program had developed an ultrasonic technique to externally test the weep holes for 2.5 mm (0.1 in.) cracks without having to enter the wet fuel tanks, as required by a handheld bolt hole eddy current technique. The initial ultrasonic method detected the cracks, but it was difficult to determine if the cracks were going up or down from the weep hole. The analysis period could be as long as 30 min per hole. The C-141 production testing team required a faster technique. Working with Northwestern University, a neural network was developed to quickly analyze the weep hole to detect a 1.8 mm (0.07 in.) crack and to determine if that crack was located in the top or the bottom of the hole. When it came time to run the POD demonstration, two PODs were conducted. The first, using only the operator's analysis to determine the presence of a crack and its location, was run on existing specimens. The results validated that a 1.8 mm (0.07 in.) crack could be located and its position determined using a modified ultrasonic imaging technique. The second technique used the neural net to test the same specimens. The results validated that the same testing method could now detect 0.5 mm (0.02 in.) cracks and determine, with fewer than one percent false calls, the location of the cracks. This proved that a greater than 3 to 1 reduction in crack length was detected using a neural net. This is a significant improvement compared to the results obtained by a trained operator. See Table 2 for a comparison of crack sizes that can be detected using a neural net versus using C-scan data.

The cost of not conducting a POD for the implementation of a new NDT technique may seem less expensive at first glance. However, when a full analysis is conducted and the problems of not having a fully validated technique are considered, the cost for the insurance that a POD brings to the overall success of any program is well worth the price. In addition, the more PODs that are conducted, the lower the overall cost, due to commonality of specimens, frames to hold the specimens, modifying procedures (rather than developing them from scratch) and the advantages one gains from having experienced personnel with whom to accomplish the validated process.

When engineers estimate the cost to implement a new technique, the equipment is typically assumed to be the most expensive component. Thus, the acquisition costs of the equipment are a major driver when making a selection. The actual costs of fully implementing almost any NDT technique usually prove that the purchase price of the equipment is only 5 to 10 percent of the total cost. To demonstrate why this is true by comparing the POD steps to a non POD program, Table 3 depicts the steps required for a non POD NDT task implementation.

What are the costs of missing a critical discontinuity? What are the risks of not conducting a POD demonstration? With any test program, there is always a risk of not finding a potentially critical discontinuity. Most NDT procedures in use today were not validated with an independently conducted POD. Some could argue that those types of procedures have brought us this far and that changing is unnecessary. If all issues were equal and we were not flying, or planning to fly, the aircraft as much as two to three times their original design life, the old ways may be sufficient. However, things are not equal. In addition to flying the aircraft well beyond its design life, some aircraft designs and materials are different than others. Some aircraft have been maintained better, some have seen different service, some have had their missions changed many times and many have flown in radically different environments than others. Great efforts are being made by leaders of the aircraft structural community to assess their aircraft. Airframe structural integrity program managers, airframe manufacturers, structural engineers and aircraft owners continually reassess their aircraft to insure that they are flying safely. The NDT community must do its part in providing the best methods within its technical abilities to insure the integrity of the NDT techniques. What are the costs of missing a critical discontinuity? What are the costs of allowing a high false call rate? What are the costs of implementing the wrong NDT technique? These are the questions that can be answered with much higher integrity if a POD demonstration is conducted to validate all phases of a critical test of any structure. When the final numbers are analyzed, it has been proven that these validated POD techniques do a much better job of testing for anomalies. In addition, they significantly lower the testing cost, saving millions of dollars and untold numbers of lives over the useful life of the aircraft.

 

Conclusion
This paper has shown that the POD process validates the NDT technique and describes the steps required to implement the application of those NDT techniques on the aircraft. The upside of conducting the POD demonstration is the rapid transfer of the technique to the production line. However, if a POD is not conducted, the following risks may result: not finding a critical discontinuity, having a system that produces a high rate of false calls or applying the wrong technique in the first place. Any one of these risks could conceivably carry a much greater cost than if the POD was conducted correctly in the first place. Almost all of the foreseeable pitfalls have been considered during the execution of the POD. Testers have been trained and made part of the solution. Calibration blocks have been certified, procedures tested and approved, equipment and software fully tested and the entire process has been validated within the probability and confidence levels
required by the structural engineers. They can use the data obtained to continue to safely fly their aircraft. Most significantly, the new techniques use more and more automated equipment operated by lower cost computers. The new NDT techniques are capable of finding smaller discontinuities with more consistency, giving the structural engineers a more accurate method of predicting discontinuity growth. This data extends the time required between tests, which greatly
reduces costs. Thus, the new process requires fewer hours of testing time, more time between tests and produces more accurate and repeatable data. The POD demonstrations are a win/win solution for the engineers who are willing to put in the effort required to accomplish them correctly and their sponsors, who own aircrafts that remain capable of safely flying for many years in the future with reduced testing costs.

 

Acknowledgments
The author is grateful to the various program managers who provided input, support and encouragement on PODs, past, present and future. These include Tommy Mullis, NDI program manager at WR-ALC/TIE NDI; Russell Alford, C-141 chief engineer at WR-ALC; Edward Pratt, C-130 ASIP manager, WR-ALC; Raymond Waldbusser, C-130 Navy liaison manager, WR-ALC; Alex Gaskin, C-141 ASIP manager, WR-ALC; Gavin Evans, B-1B bomber ASIP manager, OC-ALC; John Brausch, AFRL Materials Lab — B1-B Support; and Charles Buynak, AFRL hidden corrosion program manager.

Contributors to the paper include Matt Golis, president of Advanced Quality Concepts, who was involved with the creeping wave transducer used on the C-141 weep/rib clip holes; Richard Binder, team leader for the B1-B bomber structures at Boeing/Long Beach; Bob Bell, program manager for the Corrosion Fatigue Structural Demonstration at Lockheed/Marietta; Jan Achenbach and John Aldrin, who teamed on the neural net development for C-141 weep holes, rib clips and the C-130, from Northwestern University; and Floyd Spencer, of the Sandia National Laboratory, who has completed six of our PODs, has three in process and three in the planning stages.

Further, the author wishes to thank SAIC staff who worked on various phases of the PODs over the last ten years to support the work at Ultra Image, including Bart Drennen, Charles P'an, Tim MacInnis, Nick Nichols, Dean Christie, John Mandeville, David Judd, Chris Trey and Glenn Andrew. Finally, thanks to both Eric Lindgren and Jacqueline Princevalle for their editorial advice.

 

References
Federal Aviation Administration, Reliability Assessment at Airline Inspection Facilities, Volume 1: A Generic Protocol for Inspection Reliability Experiments, DOT/FAA/CT-92/12, 1 FAA Technical Center, Atlantic City International Airport, New Jersey, 1992.

US Air Force Aeronautical Systems Center, Military Handbook, MIL-HDBK-1823, Nondestructive Evaluation System Reliability Assessment, 1999.

 

* SAIC/Ultra Image International, Two Shaw's Cove, New London, CT 06320; (860) 442-0100; fax (860) 442-2389; e-mail <grillsr@saic.com>; Web site <saic.com/products/inspection>.

 

Copyright © 2001 by the American Society for Nondestructive Testing, Inc. All rights reserved.

[ Materials Evaluation ]

IRRSP, NDT Handbook, The NDT Technician and www.asnt.org are trademarks of the American Society for Nondestructive Testing, Inc. ACCP, ASNT, Level III Study Guide, Materials Evaluation, Nondestructive Testing Handbook, Research in Nondestructive Evaluation and RNDE are registered trademarks of the American Society for Nondestructive Testing, Inc. ASNT exists to create a safer world by promoting the profession and technologies of nondestructive testing.