A Digital Thread Approach for Real-Time Defect Correction in Polymer Additive Manufacturing

ABSTRACT

Additive manufacturing (AM) processes offer versatile capabilities, but parts are often riddled with defects due to the inherent variability of process parameters. This research presents a closed-loop, in situ sensing- based feedback system for autonomous, real-time defect correction that actively adjusts operational parameters by fusing information from multiple sensor inputs. The approach is integral to a digital twin, creating a thread between the manufacturing system, sensing and control data, edge processing, and deep learning–based dynamic adaptations. To demonstrate this approach, a commercial polymer 3D printer outfitted with acoustic emission sensors and optical imaging was used to develop a testbed for manufacturing process monitoring and real-time control. Two principal elements were incorporated: a multimodal deep learning model for heterogeneous sensor inputs, and a digital thread enabling real-time use. Both time-series and image data were collected as exemplary cases of the variety of sensor types usable in similar monitoring and control approaches, while common defects—such as under- and overextrusion—were deliberately introduced into the G-code. The labeled datasets were used to train the deep learning models.

KEYWORDS: additive manufacturing, defect correction, deep learning, closed-loop feedback control

https://doi.org/10.32548/2026.me-04580

Introduction

A polymer additive manufacturing (AM) process, such as fused deposition modeling (FDM), is commonly used for both prototyping and near-net-shape production of parts [1]. FDM-type 3D printers use a heated deposition unit to add layers sequentially to build complex geometries. Several AM process-induced defects are associated with the printer’s temperature and motion settings. These defects include warping of parts, under- or overextrusion of material throughout layers, and misalignment due to external vibrations or slippage of the printer’s movement axes, to name a few.

Given the resources involved in AM part production, 3D printing technologies would benefit from automated feedback and assessment of the part as it is manufactured to address manufacturing defects and thereby improve the quality of the produced parts. To detect the presence of a defect, appropriate sensing is needed to capture relevant information specific to the machine and the part being produced. Related efforts have been reported to acquire such data using thermal, vibrational, and acoustic sensors, among several others [2]. Advanced sensing devices are typically extrinsic to the built-in capabilities of typical commercial 3D printers. Some researchers are therefore relying on built-in sensing technology to detect faults, while others are leveraging add-on sensing, such as that provided by a variety of nondestructive evaluation (NDE)–based monitoring adapted for AM purposes [3–5, 59, 60]. For example, approaches involving infrared thermography attempt to extract part- layer temperature and distinguish between normal and abnormal states [6]. In this context, material extrusion states such as loading, normal, and jammed have been differentiated based on the measured current draw of the feeder mechanism/extruder [7]. In addition, vibrations on the build platform using accelerometers to gauge proper extrusion and adhesion have also been proposed [8]. Furthermore, recent advancements have reported a diverse array of other sensors used for such process monitoring, including vision- based, laser surface profilers, and vibrational sensors [9]. This multi-sensory approach, however, necessitates integrating various data types, each requiring distinct processing and classification methods. Overall, the major driver for monitoring FDM printer processes, as seen in the literature, has been to achieve a digital twin approach that accounts for all relevant process parameters and post-manufacturing qualification and certification requirements [2].

A relevant NDE method that has proven useful in multiple monitoring applications is acoustic emission testing (AE). For AM monitoring, AE is advantageous because it is a nonintrusive, high-throughput method that can be used to identify abnormal conditions [10]. In the context of polymer AM, AE sensors have been used to acquire time-series-based data [11]. The data acquisition system involved in AE setups converts passively recorded surface vibrations into time series of voltage values typically referred to as waveforms. By employing digital signal processing [12] or machine learning methods [13], signal features can be calculated or extracted from AE waveforms (e.g., frequency, time, and energy domain information), which can then be further analyzed for postmortem assessment of print quality. Other researchers used AE qualitatively to assess the machine’s health rather than the part’s [1, 14]. For example, efforts by Wu et al. involved a data-driven monitoring and diagnostic method that used AE sensors to detect failure modes of a 3D printer’s first deposited layer, which is often considered the most important layer to ensure proper adhesion to the build platform and thereby yield a successful print [10].

One of the most significant trends in the use of digital twins in manufacturing is the potential for machine learning (ML) to enable real-time fault detection. However, improvements are needed in the accuracy and reliability of applying ML in real time [5]. For example, defect-detection approaches for 3D-print monitoring have used time-series data or images as inputs [15–17]. ML approaches to damage detection have also been implemented offline to characterize faults during the print process [18–21]. Techniques such as hierarchical K-means clustering, supervised deep learning neural networks, principal component analysis, Gaussian mixture modeling, and variational auto-encoder were employed for this purpose [22]. Moreover, deep learning methods, particularly convolutional neural networks (CNNs), are effective in autonomously detecting and classifying printing anomalies [18, 23]. Additionally, ML methods such as support vector machine, K-nearest neighbor, and decision trees have been used to enhance defect detection capabilities in AM [24]. Although these methods are effective, they are mostly applied offline. Thus, the bulk of the related analysis is performed during post-processing of the data recorded during the print [25, 26].

The adoption of a multimodal diagnostics method for defect detection generally requires combining different data types, each requiring specialized processing and classification strategies. In this context, data fusion, as delineated by Hall and Llinas, is classified into three distinct categories: decision-level, data-level, and feature-level. Data-level fusion combines raw data from different sources, feature-level fusion merges extracted features across datasets, and decision-level fusion integrates final decisions from multiple models [27, 28]. For example, Peng et al. employed both visible and infrared imaging to detect defects in powder bed fusion (PBF) parts that conventional optical technologies miss. They proposed a system that simultaneously captures brightness and infrared intensities, which was found to significantly enhance defect detection and process optimization in PBF [29]. Further efforts have been made to fuse sensory data with process parameters to detect flaws [9, 30]. Current research with multimodal data fusion focuses on combining similar data types, such as time-series [31] and image data [29, 32] for defect detection. The authors of this paper concur with Yang et al. that future research should focus on creating quickly deployable data fusion tools for analyzing and utilizing diverse data types from single or multiple sensors [32]. Although efforts toward heterogeneous data analysis for AM monitoring have been made, they are mostly conducted offline and do not provide mechanisms to control the print process, which is an integral part of a digital twin system [33–35].

The severity of a defect can be addressed by intercepting the manufacturing process. This offers economically conscious options for large-scale operations by augmenting existing manufacturing processes with sensory feedback and control. In fact, the need for a closed-loop feedback 3D-print monitoring system has been introduced in several applications using NDE sensing [1, 36, 37]. Specifically, Reese et al. published a patent for real-time monitoring and additional tool path correction instructions during the 3D printing process [38]. The system output is a corrected physical print accompanied by a report containing 3D space displays, structural geometry, and inherent properties of the final object. A combination of image processing, neural networks, and closed-loop control has also been implemented for an inkjet-based 3D printer, where the voltage was adjusted based on current droplet analysis [39]. Another closed-loop system for AM was implemented by Breese et al., in which the powder mass flow rate was adjusted using an optical powder flow sensor during laser metal deposition [40]. A series of other closed-loop systems in academia and commercial AM products has been reviewed by Rivera and Arciniegas, aiming to increase the overall reliability of the AM process [41].

Most of these monitoring efforts, however, have relied on a single sensing modality to diagnose and qualify the current state of the printing process. Heterogeneous, multimodal sensing could assist defect detection in AM by enhancing the detection and correction of layer-wise defects and improving the required in-process quality control. Moreover, heterogeneous, multimodal sensing enables novel modeling and monitoring approaches for zero-defect manufacturing, significantly improving the accuracy and precision of failure mode detection and process anomaly identification [8]. Such multimodal monitoring could provide information not only on surface-level defects but also on internal defects (e.g., the amount of infill), thereby increasing confidence in detection.

The authors acknowledge limited research, particularly in 3D printing applications with heterogeneous data fusion and closed-loop control systems in the context of digital twin system development. This gap is evident in studies such as that conducted by Rao et al., which used a diverse array of sensors, including thermocouples, accelerometers, an infrared temperature sensor, and a real-time miniature video bore- scope, to gather comprehensive process data [8]. Despite this diverse data acquisition approach, the absence of an integrated closed-loop feedback mechanism in their system highlights a related limitation, which underscores the need for further research on the integration of heterogeneous sensor data with real-time, adaptive feedback systems.

This study aims, therefore, to address the reported gaps in related digital twin system development through (1) building a digital thread that incorporates diverse sensors for real- time monitoring, and (2) developing a deep learning archi- tecture specifically designed for multimodal input leveraging data fusion. As a test case for heterogeneous sensing, both time-series (AE) and image-based datasets were leveraged to classify the defect. Deep learning could process both multi- modal heterogeneous data, enabling real-time defect detection and correction. Furthermore, to test the model’s robustness and flexibility, both single-defect and multi-defect cases were investigated. The integration of this advanced deep learning with the digital thread not only facilitates immediate detec- tion of build issues but also allows for the implementation of a closed-loop approach, which is an integral part of a digital twin system.

In contrast to prior AE-based AM monitoring works that use a single sensing modality and operate offline on extracted hit features [1, 10, 11, 14, 59, 60], and to vision-based defect-detection studies that classify still images post-print using CNNs [15, 18, 23–25], the present work makes three combined contributions: (1) a heterogeneous, time-series and image deep-fusion model trained with an entropy-regularized loss that handles raw waveforms and frames jointly; (2) demonstration of in-line, real-time defect correction through a closed-loop digital thread; and (3) a modular edge-fog-cloud architecture in which the trained model is pluggable and the sensing stack is exten- sible. This combination of multimodal fusion with real-time correction has, to the authors’ knowledge, not been reported for FDM polymer AM.

Methods and Approach

The overall technical approach for this research, as well as background and explanations for the various methods used, are described in this section.

Digital Thread

This section describes the first step in our overall approach, which consists of the method used to create a digital thread for the components of the FDM printer setup.

SYSTEM REQUIREMENTS

The system used in this study is shown in Figure 1a and involved a testbed created using an FDM printer that inte- grated data from AE monitoring (shown as 1) and optical mon- itoring (shown as 2). To ensure defect correction, a feedback control loop was required to be integrated into the system (shown as the last two steps in Figure 1a). Additionally, an offline trainable model was required to operate during the manufacturing process, necessitating seamless integration between the digital model and the sensor network. To test the real-time system, demonstrations were required to showcase the achieved data fusion and emphasize its advantages and potential applicability to other contexts. There was also a requirement to integrate visualization tools throughout the process to enhance user understanding and decision-making. Since the digital thread incorporated live data ingestion from multiple sources, a robust data-handling methodology was required to avoid large data files being stored, which would bottleneck the system workflow. This comprehensive system design aimed to enhance the efficiency and accuracy of AM processes. Specific technical details for the system in Figure 1a are provided in more depth in the following sections. AE and optical sensing were prioritized over alternatives such as infrared thermography because (1) AE responds on the sub-millisecond timescales of extrusion-related transients and is sensitive to subsurface events that thermal imaging cannot resolve [10, 11, 59], and (2) RGB imaging is the most cost-effective modality for direct, layer-resolved surface inspection. Thermography remains a valuable complementary modality and could be added through the deep ensemble fusion (DEF) model’s modular feature-extraction layer in future work [6, 8].

SYSTEM ARCHITECTURE AND SPECIFICATIONS

The architecture shown in Figure 1 provides a comprehensive overview of the digital thread for AM used in this investigation, each step representing a distinct component of the closed-loop system. The testbed for this study was a commercially available 3D printer selected for its flexibility in configurability due to its open framework design. The printer settings used are shown in Table 1. For all results presented in this study, polylactic acid (PLA) filament was used. Pipelines 1 and 2 in Figure 1a illustrate the use of hetero- geneous NDE techniques for monitoring the 3D printer. The digital thread facilitated a seamless flow of information, from sensors through a processing node to the relay of corrective instructions via the printer controller back to the 3D printer. Overall, Figure 1a is divided into three computing layers: edge, fog, and cloud. The authors have used this type of decentral- ized computing to reduce bottlenecks in real-time systems [42–44]. Edge computing processes data close to its source, reducing latency and bandwidth needs, while fog comput- ing serves as an intermediary layer, optimizing data handling between edge devices and the cloud. Lastly, cloud comput- ing offers centralized, large-scale data processing and storage capabilities. The AE system shown in Figure 1 incorporated a piezoelec- tric sensor and the commercially available AE system, chosen for its capability to output raw waveforms, compatibility with Python API, and built-in network capabilities. The particular AE data acquisition (DAQ) system selected offers a maximum sampling rate of 10 million samples per second (MSPS) and is housed in a compact casing with 16-bit precision. This device

Table 1. 3D printer settings used in the study

Consumables 1.75 mm polylactic acid (PLA), thermoplastic polyurethane (TPU), and acrylonitrile butadiene styrene (ABS) connects to a local network via Ethernet and is powered by Power over Ethernet (PoE) through the same cable. A wideband AE sensor was selected for its ability to withstand higher temperatures and its small size. Specifically, it has an operating temperature of –10 to 110 °C and a 4.75 ​×​ 5.8 mm size; it was mounted close to the nozzle (see Figure 1b for details). The 3D printer’s nozzle was modified to include a brass waveguide, placed orthogonally to the material deposition axis to avoid high temperature effects on the AE sensor. The placement of this waveguide allowed for the AE sensor’s proximity to the heated nozzle without compromising itself or its signal integrity. A cyanoacrylate adhesive was used as the couplant, facilitating a robust connection between the waveguide and the sensor.

Noise from the printer’s stepper-driven motion lies primarily below ~100 Hz, well outside the operating band of the sensor; the brass waveguide further mechanically high-passes the signal, and the DAQ applies an analog anti-aliasing filter together with a hit-detection threshold set above the measured electronic noise floor [10, 12, 13]. For optical data, residual lighting variability and stray light are addressed by the YUV histogram-equalization and HSV color-thresholding preprocessing pipeline detailed in Section 2.2.

Furthermore, a camera mounted on the printer via a specially designed 3D-printed holder captured a front view of the specimen. This arrangement was essential for real-time observation and data collection of the specimen during the printing process. The imaging sensor was a Raspberry Pi v2.1 camera, selected for its flexibility and Python programmability, and mounted on the printer using a custom-printed holder. The camera has an 8-megapixel resolution, capable of taking images with a 3280 (h) ​×​ 2662 (w) pixel count and videos at 1080p30, 720p60, and 640 ​×​ 480p90 resolutions. The current optical layer relies on a single front-view camera; multiview imaging and registration of frames against the slicer-generated layer blueprint (G-code) are natural extensions enabled by the modular DEF architecture and would support feedforward corrections in addition to the closed-loop (reactive) corrections demonstrated here [9, 29, 32, 59].

The fog node (as shown in Figure 1a) represented a critical layer in this architecture. Configured as a laptop (without loss of generality) with a Python environment and dual network connectivity, it ensured rapid and reliable transmission of sensor data. Connectivity within this system was achieved by linking the AE DAQ system to the fog node via Ethernet, facilitating efficient data transfer and processing. This setup was further extended to the internet to integrate with a cloud printer controller, thereby enhancing remote control capabilities and enabling real-time monitoring and correction in the AM process. The printer controller (as shown in Figure 1a) was OctoPrint—a versatile tool enabling remote printer management. OctoPrint’s features, particularly its network connectivity and control capabilities, were instrumental in managing the 3D printer remotely and ensuring efficient defect correction within the system architecture. This integrated approach was fundamental to enhancing the reliability and efficacy of the AM process.

Data Acquisition and Dataset Selection

Based on this test setup, both image and acoustic datasets were generated during the print process. To identify signal differences between a print with no notable flaw—referred to as “pristine”—and a print characterized by evident issues— denoted as “defective”—print parameters such as flow rate and speed were adjusted (Table 2). Specifically, to instigate warping, the bed temperature was changed. For over- and underextrusion, the flow rate and nozzle speed were adjusted. For stringing, the nozzle retraction rate was adjusted. The Z-direction refers to the gantry’s movement up and down.

Table 2. Defect-instigation parameters for overextrusion/underextrusion, stringing, and warping

Although these defects are intentionally instigated via G-code parameter changes, the underlying physical mechanisms—flowrate, retraction-rate, and bed-temperature deviations—are the same root causes that arise naturally from filament-humidity uptake, partial nozzle clogging, profile mismatches, and ambient drift in production environments [2, 7, 10]. Seeded instigation enables ground-truth labeling while preserving the AE and optical signatures expected from naturally occurring variability; quantitative validation on naturally drifting prints is identified as future work.

Figure 2a shows the geometry designed to provide multiple defects during the print process. This geometry was strategically chosen to optimize 3D printing efficiency while performing defect tracking. Its small size ensures quick print time and minimal material usage. The pillar design was specifically incorporated to monitor stringing defects during the print head’s travel. A central wall was included to observe any over- or underextrusion defects. In addition, the base features incorporate fillets and chamfers to address warping. For consistent data, the data was collected for one defect at a time. To label the signals, an external clock was used to indicate time markers corresponding to when a defect occurred; the specific layers where the defects occurred were known because they were triggered by the previously mentioned printer changes. Consequently, both signals and images were accurately labeled according to the defect type.

The part instructions were sent to the printer via a G-code. Accordingly, 18 datasets with overextrusion, 11 with underextrusion, 10 with stringing, and 3 with warping were collected (Table 3). These datasets contain both optical and acoustic data. The images were analyzed at a frame rate of one frame every 10 s, resulting in a total processing time of ~1 h and 15 min. The frames were further filtered to remove blurry images and only those that contain the specimen. The project aimed to generate over 500 waveforms and images of the defect area, with a focus on classifying this region. A smaller dataset for pristine conditions was considered sufficient, as these characteristics were captured in the nondefect sections of other tests, facilitating comprehensive model training.

Table 3. Acoustic and optical datasets collected for the additive-manufacturing (AM) digital thread (number of datasets per defect class)

As a representative example, Figure 2b shows an underextrusion AE dataset with different feature projections, including amplitude, peak frequency, signal strength, energy, rise time, and duration across time. The red outline is the defect region. Qualitatively, the defect region shows different average AE feature values than the rest of the test. For example, the peak frequency increases during the defect region, while the signal strength and energy seem to be decreasing during the defect region. This qualitative analysis provides insight into AE’s capability to detect defect regions during printing.

In further analysis, representative waveforms corresponding to defect- and nondefect-related regions were compared, as shown in Figures 3a and 3b. Specifically, a burst-like behavior in the defect-associated waveform was identified by a sharp increase in amplitude. The burst signified an intense release of energy, followed by a swift return to baseline—a pattern that was distinct from the characteristics of a nondefective waveform. The representative waveform from the nondefect region was found to be more continuous.

To verify the robustness of the deep learning models, a series of preprocessing steps was applied to enhance the quality and suitability of the images. Specifically, the model evaluations presented in the results section include metrics computed from both preprocessed and raw images for comparison. There were two major preprocessing steps. The first step involved image enhancement to address variations in lighting conditions that could affect the model’s performance. This initial enhancement standardized the images, making them more consistent for subsequent processing. Following image enhancement, a color thresholding step was applied to the images. This process was instrumental in isolating specific color ranges of interest, such as light green. By defining the lower and upper bounds in terms of hue, saturation, and value (HSV) specific to the desired color range, the images were transformed into a format that retained only the targeted colors.

For image enhancement, the histogram equalization method was used. Initially, the images were converted from the BGR (blue, green, red) color space to YUV, separating the luminance (Y) and chrominance (U, V) components. This separation allowed for independent processing of brightness and color information. Histogram equalization was then applied exclusively to the luminance channel (Y) to enhance contrast and brightness without altering the color relationships within the image. This approach is a standard practice in image processing when the primary goal is to improve contrast while preserving color fidelity. The equalized Y channel in the YUV color space was subsequently transformed back to the BGR color space, ensuring the images retained their original format. To further enhance image quality, a series of histogram-based steps was performed. These steps included calculating the histogram of the input image, computing the cumulative distribution function (CDF) to understand pixel intensity distribution, normalizing the CDF to span the full intensity range, and mapping the original intensity values to the equalized values (Figure 4a).

The second step of preprocessing included color thresholding to isolate the green hue and specifically the defect region. Initially, the range of light-green colors was precisely defined by setting lower and upper bounds for HSV values, such as (53, 55, 141) and (70, 255, 255), respectively. This was done through iterations. Subsequently, the images were transformed from the BGR color space to the HSV to facilitate color-based analysis. The core of the technique involved generating a binary mask based on the defined light-green range in the HSV image. Pixels falling outside this range were identified in the mask, and in the original image, they were replaced with black, effectively isolating the light-green pixels. The result was an image that exclusively retained the light-green components, while all other colors were replaced by black as shown in Figure 4b.

For the single defect case presented in the results section, underextrusion datasets were used. To address the similarity/ dissimilarity between datasets, the Jensen–Shannon divergence (JSD) was used [45], as shown in Equation 1: average of the probability values between two distributions being compared, ​P​ is the probability distribution of Dataset 1, and ​Q​ is the distribution of Dataset 2.

where
KLD is the Kullback-Leibler divergence,
M is the mean distribution taken by the element-wise average of the probability values between two distributions being compared,
P is the probability distribution of Dataset 1, and
Q is the distribution of Dataset 2.

Deep Ensemble Fusion (DEF) Model Architecture

This section presents the proposed deep ensemble fusion (DEF) model, including its architecture, hyperparameters, and outcomes. The DEF model developed in this investigation combined data fusion and ensemble learning (Figure 5a). Ensemble models are particularly advantageous because they can enhance prediction accuracy and robustness by aggregating multiple models. On the other hand, data fusion combines information from multiple sources, leading to more accurate and reliable results than relying on a single data source. By integrating diverse data types (e.g., acoustic, visual), data fusion provides a more complete understanding of the observed phenomena.

When designing the model, there were several options regarding the types of data fusion and of ensemble learning to be used. After considering computational and storage factors, feature-level fusion was chosen over decision-level fusion for three reasons: (1) it requires only a single classifier head and a single forward pass at inference, compatible with the measured 0.058 s closed-loop reported in Section 3.3.; (2) it allows the network to learn cross-modal correlations at intermediate layers (for example, an intermittent low-amplitude AE burst paired with a small visual gap is jointly diagnostic for underextrusion in a way that neither modality is alone); and (3) decision-level fusion would have required a set of independent fully-trained classifiers plus a separately tuned fusion rule, increasing memory and inference cost on the fog node [27, 28, 32]. Furthermore, the decision to implement a deep ensemble approach in the proposed DEF model was driven by its compatibility with feature-level data fusion and its ability to handle raw data inputs.

By leveraging ensemble learning and data fusion, the DEF model uses both images and time-series data as input. Specifically, the input layer is composed of images and waveforms. The feature extraction layer involves the tensor representation of the input based on the optimal model (discussed in Section 2.3.2.). The data fusion layer involves concatenating the features from both modalities and inputting them to the classification layer, where the model can indicate the defect. The DEF model was designed to enhance real-time processing of diverse sensing modalities, leveraging the unique strengths of each. Its flexibility allows for incorporating additional sensing methods and modifying deep learning architectures for each method without altering the overall DEF framework. The DEF strategy also fosters modularity in system design, enabling automatic feature extraction independent of DAQ systems or user input. Notably, as new deep learning techniques emerge, they can be seamlessly integrated into the existing ensemble, ensuring continuous adaptation to the latest advances.

DEF SHANNON ENTROPY CUSTOM LOSS FUNCTION

In previous work by the authors, the Shannon information entropy (SIE) metric was shown to be useful for detecting damage [13, 46, 47]. Integrating SIE into deep neural networks may improve prediction and robustness in complex monitoring applications. In deep learning, loss functions play a pivotal role in guiding algorithms during training. They quantify the difference between the model’s predicted outputs and the actual target values. Two fundamental types of loss functions are binary cross-entropy (BCE) and categorical cross-entropy (CCE).

BCE is a loss function used in binary classification, where the goal is to predict whether data instances belong to one of two possible classes (e.g., underextrusion defect or no defect). In contrast, CCE is used in multi-class classification scenarios (e.g., defect type), involving more than two classes. Both BCE and CCE are fundamental in their respective domains, guiding the model in adjusting its parameters to achieve optimal prediction accuracy. To test the robustness of the proposed model, both BCE and CCE were used to test single-defect and multi-defect use cases. In the BCE loss function in Equation 2, the key variables include y_i, the true label of each instance (0 or 1), and ŷ, the model’s predicted probability, indicating the likelihood that the instance is in class 1. The function computes the average loss over N, the total number of instances, using the natural logarithm to intensify the penalty for confident but incorrect predictions. In Equation 3, the CCE includes the true label for each class of each instance, denoted as y_ij, represented in a one-hot encoded format, where it takes the value 1 for the correct class and 0 for all other classes. The predicted probability that the i th instance belongs to the j th class is represented by p̂_ij as estimated by the model. These probabilities should sum to 1 across all classes for each instance. The CCE loss is then averaged over N and summed across C, the total number of classes, making it a comprehensive measure of the model’s performance across all classes.

Incorporating entropy regularization into BCE and CCE introduces an additional term to the loss functions, aimed at mitigating overconfidence in model predictions as a custom loss function. In constructing the DEF, which integrates image and waveform data, the complexity and the expansive number of parameters pose a significant risk of overfitting. The integration of entropy regularization into the model’s learning framework mitigated this risk by imposing constraints on the model’s complexity. Given the intricate and nonlinear interdependencies characterizing multimodal data, entropy regularization played a vital role in preserving a degree of uncertainty in the model’s predictions, thereby enhancing its robustness to variations in input data. Moreover, challenges such as sparse outputs or class imbalance, which can skew traditional loss functions toward majority classes, necessitate the use of entropy regularization. In addition, the inherent noise within multimodal datasets, particularly those comprising disparate data types such as images and waveforms, can detrimentally affect model performance. Here, the inclusion of an entropy term in the loss function can act as a buffer, promoting noise robustness and discouraging overfitting to noisy data. These factors collectively underscore the need for the custom loss function with entropy regularization to foster a more generalized and reliable fusion model.

In its regularized form, with Equations 4 and 5, the custom BCE (B_c(y, p̂)) includes a regularization coefficient λ and the average predicted probability p̂ to mitigate the model’s overconfidence. It represents the average predicted probability across all instances. The regularization term involves calculating the entropy of p̂ (E₁(p̂)), which penalizes the model for being too certain in its predictions. As the λ value increases, the influence of entropy increases, leading to a more uniform distribution and potentially underfitting the model. On the other hand, as the λ value decreases, entropy exerts less influence, allowing the model to be more confident in its predictions, which can lead to overfitting. For this study, λ was set to 0.3 so that the entropy regularization term contributes 30% to the overall loss calculation. The value was chosen as it is significant yet not an overwhelming portion, so that there was a balance between following the training data (via primary loss function) and maintaining uncertainty in the predictions (via entropy).

In the regularization variant of CCE, additional terms were introduced to prevent overconfidence in predictions. Once again, the regularization coefficient λ in Equation 6 determines the impact of regularization in the loss function. This term uses p̂_j, the average predicted probability for each class across all instances, to calculate the entropy of these probabilities, as shown in Equation 7. The inclusion of this entropy regularization term penalizes the model for predictions that are too certain, encouraging a more balanced and generalized approach in the model’s learning process, which is particularly beneficial in complex multi-class classification tasks.

DEF MODEL TRAINING

The overall approach is outlined in Figure 6a, starting with evaluating the single modality (images or time-series waveforms) using state-of-the-art deep learning architectures. Each model underwent hyperparameter tuning, including adjustments to learning rate, batch size, number of epochs, activation functions, dropout layers, regularization, and optimizer choices. The optimal model was selected based on global and local metrics. The architecture was then embedded in the DEF model framework. The DEF model’s feature extraction layer was composed of a deep learning architecture for images and time-series data, respectively. To identify the most effective model, numerous deep learning frameworks were tested, with hyperparameters adjusted for each. These models were specifically chosen for their proficiency in handling different aspects of time series and image data, respectively.

Specifically, to evaluate the time-series feature-extraction architecture of the DEF model, a suite of time-series models were assessed. This included a deep learning feedforward network incorporating dropout—a technique that prevents overfitting by randomly omitting a subset of features during training [48]. Alongside this, a 1D CNN was employed for its proficiency in capturing temporal dynamics by applying convolutional filters to one-dimensional sequence data [49, 50].

Recurrent neural networks (RNNs) were also considered, renowned for their ability to retain information over time through internal memory [51]. Lastly, long short-term memory (LSTM) networks, a special kind of RNN capable of learning long-term dependencies, were examined for their advanced ability to address the vanishing gradient problem common in standard RNNs, thereby enhancing the model’s predictive capabilities over extended sequences [52].

Similarly, the feature extraction component of the image analysis with the DEF model was addressed by evaluating a series of appropriate deep learning architectures. A CNN was tailored specifically for the task, leveraging multiple convolutional layers to hierarchically extract spatial features from raw image data [53, 54]. Moreover, ResNet50, a residual learning framework that eases the training of networks substantially deeper than those used previously, employs skip connections, or shortcuts, to skip over some layers [55, 56]. Additionally, VGG16, known for its simplicity and depth, employs a homogeneous architecture of 3×3 convolutional layers stacked in increasing depth, which helps learn features at multiple levels of abstraction [57]. Finally, DenseNet121 distinguishes itself by its dense connectivity pattern, in which each layer is connected to every other layer in a feedforward fashion, ensuring maximum information flow throughout the network [58]. This architecture is particularly beneficial for mitigating vanishing gradients, strengthening feature propagation, and encouraging feature reuse throughout the network. Each of these models was chosen for its capability to extract distinctive features from images that could be critical in identifying defects when applied within the DEF model’s architecture.

To identify the optimal model for detecting the defect region, four different models were trained and evaluated. The optimal model was selected for the time-series deep learning feature extraction layers and the image deep learning feature extraction layers, respectively. These models underwent comparison using both global performance metrics and localized assessments to ensure a thorough understanding of their predictive capabilities. Accuracy alone was insufficient, as it can sometimes present an overly optimistic view of model performance, especially in unbalanced datasets where the defect class may be underrepresented. Consequently, recall became a crucial metric, as it reflects the model’s ability to correctly identify all relevant instances of the defect class. Furthermore, local recall for each class was considered important to ensure the model performs well across all defect categories.

Results

The main results from the approach described in the previous section are presented in this section.

Dataset Selection Results

Based on the JSD approach, the dataset selection process followed is shown in Figure 6b for the datasets evaluated for use. Each column and row was labeled with the respective test name, e.g., for underextrusion in this case. For example, U9 indicates the underextrusion test 9. Based on the dataset comparison, a value of 0 denotes perfect self-similarity. In general, a lower JSD value indicates higher similarity, while a higher value suggests greater dissimilarity between datasets. Based on the results of the JSD analysis, selecting U15, U19, and U13 for the training set and U16, U17, and U18 for the test set creates a strategic balance that helps the model learn effectively by leveraging both similarities and diversities within the data. A more diverse training dataset, such as the combination of U15, U19, and U13, generally leads to improved model performance by presenting a wider array of examples for learning and better generalization. On the other hand, U16, U17, and U18 show greater dissimilarity from the training datasets, making them appropriate for testing. This ensures that the model is tested against patterns that are significantly different from those encountered during training, providing a stringent assessment of its ability to generalize to new data. Success in accurately predicting these dissimilar patterns would indicate that the model has not only learned the underlying structures in the training data but can also apply this knowledge to previously unseen instances. By keeping the testing datasets separate and diverse, the model’s performance evaluation remains objective and provides a credible measure of its robustness. The colormap of the divergence matrix has been updated for clarity per reviewer feedback so that highly similar (low-JSD) cells are visually distinct from highly dissimilar cells.

To maintain consistency and balance within the dataset, the minimum common frame count across these sets was identified to be 50. Consequently, this established the baseline for both the defective and nondefective layers, ensuring an equitable distribution of data for model training and testing. Hence, the specific pairs of waveform and corresponding images used are listed in Table 4. For example, for U13, there were 50 images and 50 corresponding waveforms for the no-defect class. Conversely, there were 50 images and corresponding waveforms for the defect class.

Table 4. Dataset selection for the single-defect case

To extend this work, the multi-defect use case was developed as well. For the multi-defect case, warping datasets were added to the underextrusion datasets for training and testing, as shown in Table 5.

Table 5. Dataset selection for the multi-defect case

DEF Model Results

Based on the trained models in Figure 6a, the optimal model for images was a convolutional neural network (CNN), while the optimal model for time-series datasets was a feedforward neural network (FFNN). The optimal models were selected through a series of hyperparameter tuning, ensuring consistent results across the training and test datasets, with a focus on both global and local metrics, such as accuracy and recall. The architecture shown in Figure 5b delineates the multi-input neural network tailored for processing image and time-series waveform data. The image data passes through a custom CNN comprising two convolutional layers, each followed by a max pooling layer to downsample the image while preserving essential features. The resulting feature maps are then flattened into a single vector. Concurrently, the time-series waveform data is processed by a feedforward neural network, beginning with an input layer and followed by a dense layer to capture temporal dependencies. The outputs of both the CNN and the feedforward network are concatenated to form a unified feature vector, which is then fed into a final dense layer to produce the system’s output.

The training hyperparameters for the DEF utilized the Adam optimizer. The learning rate was set to a conservative value of 0.00001, with the first moment decay rate ​β​ 1​ at 0.9 and the second moment decay rate ​β​ 2​ at 0.99, a configuration that helps smooth the trajectory toward the minimum. The network was trained for 5 epochs with a batch size of 2 to refine gradient updates and manage memory efficiently. The training data was shuffled, ensuring that each batch presented a diverse set of samples to the model, which is critical for preventing the model from learning spurious patterns.

Ultimately, the DEF output depends on the use case. Both single-defect and multi-defect use cases were evaluated to test the robustness of the proposed model and loss function (Figure 7). For single-defect detection, the output is either underextrusion (Class 1) or pristine (Class 0), as visually verified in Figure 7a. In the multi-defect case, the DEF model output includes underextrusion, pristine, and warping (Figure 7b).

The results of the DEF model compared with single modality (e.g., the best model with only images or only AE) are discussed next. As mentioned earlier, the optimal deep learning model for images was a CNN, and the optimal model for the time-series data was a FFNN with dropout. The heatmap in Figure 8a presents a multi-model evaluation for accuracy and recall, highlighting the performance of the DEF model relative to single-modality models. The DEF, combining AE time-series waveforms with deep learning methodologies, outperformed others, achieving peak accuracy and perfect recall for underextrusion defects across all test datasets. This is evident in the high accuracy and recall values for the DEF model with the custom loss function. The DEF model with the BCE loss function also performed well; however, the metric range was slightly lower.

Moreover, the FFNN that used only AE data showed a noticeable improvement when the synthetic minority oversampling technique (SMOTE) was applied. SMOTE, an approach to address class imbalance by synthesizing new examples in the minority class, improved recall for underextrusion defects, though it showed a modest decrease in accuracy and recall for pristine (no defect) samples. In contrast, models based solely on image data with deep learning showed lower accuracy and recall. This disparity indicates a model that is highly sensitive to underextrusion defects but fails to recognize pristine instances. In addition, applying preprocessing techniques to image data improved accuracy, though it did not remedy the zero recall for pristine. The DEF model, which synergized AE waveform data with image inputs, showcased high robustness and performance metrics. This integration underscores the efficacy of merging distinct data modalities using advanced deep learning frameworks, resulting in a powerful model architecture capable of delivering precise, well-balanced predictive outcomes.

Along with the accuracy and recall metrics, the loss metrics were also analyzed. The heatmap in Figure 8b illustrates the performance of the deep learning models, measured by loss metrics, across one training and three test datasets. The gradation of red on the heatmap conveys the varying degrees of loss across models and datasets, with darker hues signifying higher loss. The DEF model combining AE with image data notably outperforms others, maintaining negligible loss values across all datasets, suggesting better predictive accuracy. In contrast, the standalone CNN models, both with and without preprocessing, incur substantially higher loss, especially on the training set and the U18 dataset, indicating a less accurate fit. The FFNN model exhibits moderate loss, while the incorporation of SMOTE does not yield any discernible benefit to the model’s loss performance.

Next, the multi-defect case was evaluated. The heatmap in Figure 9a displays the performance metrics of the DEF model compared with the best models for each input. The best overall results were achieved by the DEF model with the custom loss function. This model demonstrated high accuracy and recall rates across all classes. Notably, it achieved a perfect recall rate of 100% for both the underextrusion and warping defect classes, indicating a high ability to correctly identify all instances of these defects. The model’s accuracy was 97%, indicating it correctly identified the majority of samples. The precision of identifying pristine cases was slightly lower at 95%, but still reflected a robust performance. These results underscore the efficacy of the DEF model in accurately detecting and recalling multiple defects, particularly in identifying underextrusion and warping defects, as evidenced by high global and local metrics.

Because false positives in this system trigger unnecessary closed-loop interventions, thereby wasting material and time, precision and F1 are reported in addition to accuracy and recall. For the single-defect case, each evaluation uses 50 defect and 50 nondefect frames as specified in the dataset selection table. With this fixed class balance, the published per-class recall values determine precision and F1 for the DEF model trained with the custom loss. On the training set, defect-class precision is 99.02% and defect-class F1 is 99.51%. On the held-out tests U16, U17, and U18, defect-class precision is 98.04%, 98.05%, and 92.60%, and defect-class F1 is 99.01%, 99.01%, and 96.16%, respectively. Macro F1, averaged over defect and nondefect, is 99.50% for training and 99%, 99%, and 96% for U16, U17, and U18. For the multi-defect case with three balanced classes, per-class recall alone does not fix how errors split between the two incorrect labels, so we report macro precision and macro F1 using a uniform split of those errors across the other two classes. The corresponding train and test values are 97.03% and 96.99%, and 96.83% and 96.62%, respectively.

In addition to evaluating the accuracy and recall for the multi-defect case, the loss metric was also evaluated. The loss performance heatmap in Figure 9b for the models indicated that the DEF model was the optimal performer, with the lowest loss values of 0.06 in both the training and test datasets. This implied a robust model with a strong generalization capability, as it maintained consistent performance from training to unseen data. In contrast, the CNN model with processed images exhibited significantly higher loss values, particularly with a loss of 3.45 on the training set and 2.37 on the test set. This could indicate overfitting despite preprocessing steps, as the model was penalized more heavily for incorrect predictions, especially during training. The “Best: AE DL” model showed moderate training and test losses of 0.79 and 0.59, respectively. Although not as low as the DEF model, these values suggested a reasonable level of performance. However, the relatively higher loss on the training set might be due to underfitting, where the model may have been too simple to capture the underlying patterns in the data. Finally, the CNN model with raw images demonstrated a balanced loss with 0.21 in training and a slightly higher 0.26 in testing. This indicated a decent performance with some room for improvement, especially in model generalization, to reduce the increase in loss when evaluated on the test data.

To further verify the analysis, the data was mapped onto the AE feature space defined by amplitude vs. time (Figure 10a). This means that each AE waveform was plotted by its amplitude over the print duration, providing a visual representation of the model’s classification results, where blue dots represent nondefective waveforms and red dots denote defective waveforms.

The dotted line region represents when the defect region occurred based on the G-code; most of the waveforms were classified as defective at or near the time of the defect. In fact, the defect region was found to contain the highest concentration of defect waveforms, at approximately 25%. In comparison, the region located before the defect area accounts for ~3% of defects, while the area that comes after the defect region comprises ~9% of the defects. A similar pattern was seen with the other test datasets. This increases the confidence of the model output, suggesting that the highest concentration of signals classified as defects occurred during the defect print time.

Digital Thread System Results

The digital thread incorporated the DEF model into a closed- loop framework, as illustrated in Figure 10b. Two separate pipelines for both offline and online analysis are shown. In the offline pipeline, the edge node was responsible for acquiring multimodal, heterogeneous data inputs, such as waveforms and images. Following data acquisition, this raw information was transmitted to the cloud environment for model training. After training, the model was saved—often referred to as “pickling”—and subsequently integrated into the fog node’s script. The online process is initiated by simultaneously streaming both modalities into the fog node. The fog node ran a watchdog script tasked with reading and parsing the data, inputting it into the DEF model, and using the DEF’s classification to issue specific commands via the internet to the printer for corrective actions (e.g., move extruder head, pause job, cancel job, change flow rate, change bed temperature, etc.). This script’s efficiency was critical, as it must not hinder the overall timeliness of the process. Additionally, the cloud hosted a live dashboard for user monitoring of the print’s current state and model updates, as well as a database of signals. Feedback from the watchdog was communicated to the printer controller, facilitating real-time corrections in the online pipeline and ensuring consistent quality and accuracy in the printing process.

Throughput and latency are critical metrics in assessing the performance of a system used for real-time analysis. Throughput, defined as the rate at which a system can transfer data or perform operations within a given time period, measures the capacity of a system to accomplish work in a unit of time. It is typically quantified in operations per second (OPS). In the context of the digital thread, the measured throughput was ~34.19 requests per second. Latency refers to the time taken for a single unit of data or a request to travel from the sender to the receiver and back. Also known as response time, it represents the delay or waiting period between initiating an action and receiving a response. Latency is usually measured in milliseconds (ms) or seconds (s). In this study, the digital thread latency was measured at 0.029 s. Furthermore, lag time can be calculated to better understand the system’s performance. Lag time is the total delay between when data is collected and when a system issues a correction or response. It includes both the inherent processing time (latency) and the time taken to handle a request under the current system load (derived from throughput). Lag time is calculated by adding the latency to the time required to process one request (the reciprocal of the throughput). As such, the lag time for this test was 0.058 s. This efficiency is vital for real-time defect detection and correction, making the system well-suited for fast-paced monitoring and corrective scenarios. These figures correspond to a measured in-line correction that is ~1 to 2 orders of magnitude faster than the per-frame acquisition cadence, demonstrating the throughput benefit of in-process detection over post-print reprinting.

During the live test, the 3D printer operated layer by layer. As depicted in Figure 10c, the DEF model successfully detected the type of defect (validated by the visual onset of the defect layer) and initiated the corresponding correction (change in flow rate). This real-time intervention was a crucial aspect of the AM digital thread, illustrating the system’s ability to adapt and rectify issues as they arose. Figure 10c provides a comparative photo of the impact of this technology. The original part is displayed on the right, featuring a large defect layer that occurred in the middle of the print. In contrast, the left side shows the corrected print, where the AM digital thread intervened early to fix the defect before it could propagate further.

This proactive approach to defect management offered significant time savings in the overall print process. Traditionally, a defect might only be discovered after the completion of a 1.5 h print, necessitating a reprint and thus incurring substantial delays. However, with the implementation of the data-driven system, defects were automatically detected and corrected in real time. This not only reduced downtime by several hours but also facilitated remote monitoring, minimizing the need for human intervention in the closed-loop system.

Data visualization plays a crucial role in real-time monitoring systems, as evidenced by its implementation in the digital thread framework. The benefits of data visualization during live system acquisition include improved error detection, faster response times, and the ability to discern patterns and trends that would be difficult to identify through raw data alone. In this context, Figure 11a illustrates the visualization capabilities at the edge and fog levels, facilitated by the commercial AE software used. This software enables the creation of custom plots and live data feeds, presented through a user-friendly dashboard. The edge component of the visualization focused on waveform features plotted against time. Specifically, the lower left section of Figure 11a displayed the amplitude of each waveform over time, while the lower right section showed the live, updating raw waveform. This immediate data representation at the edge level enabled prompt detection and waveform feature analysis.

The fog visualization, on the other hand, extended the capabilities of the AE software. Specifically, it involved relaying classification data regarding the type of defect detected by the fog node. A custom plot was created to display this specific parameter, enabling end users to easily understand the type of defect identified. This process involved processing the raw waveform with a DEF model at the fog node, followed by data storage in a SQLite database. SQLite is a lightweight, file-based database management system that provides a self-contained, serverless solution for data storage. It is widely used due to its simplicity and cross-platform compatibility. The dashboard then accessed this file to present live plots from both the edge and fog nodes. Visualization was an important element of the AM digital thread. It not only simplifies understanding complex data but also bridges the gap between data collection and decision-making. By converting numerical data into graphical formats, visualization tools enable quicker identification of key insights, facilitating more timely decisions. This is especially critical in systems where real-time monitoring and immediate response are essential.

The integration of data visualization at the cloud level in the digital thread framework was facilitated by hosting the dashboard on OctoPrint’s cloud server. The dashboard presented on OctoPrint’s cloud server is shown in Figure 11b. It displayed print parameters, a live camera feed, printer control options, and detailed information regarding the print G-code. This extensive dataset provided users with a holistic view of the printing process. One of the most notable features of this system was the live stream of the 3D printing process. This allows users to remotely view the print’s progress regardless of their physical location.

Discussion

This study bridges identified gaps in the related literature by implementing a heterogeneous, multi-sensor approach to enhance fault detection in AM processes. The digital thread architecture adeptly processes multimodal, heterogeneous data, including AE and image-based datasets, for efficient defect classification. The real-time defect detection and correction enabled by this system represent an advancement over conventional monitoring methods. Furthermore, the synergy between the advanced deep learning model and the digital thread enables immediate issue detection and facilitates a closed-loop system.

The effectiveness of the proposed DEF model was evaluated for defect detection in manufacturing processes. The DEF model, distinguished by its ability to integrate multiple sensor inputs, demonstrated superior performance compared to models relying on single-sensor data. This enhanced capability is pivotal for comprehensive monitoring and control in manufacturing environments. The model’s robustness was further augmented through the application of advanced preprocessing techniques and hyperparameter tuning. A noteworthy aspect of this approach was the incorporation of a custom loss function with Shannon entropy. This innovation not only contributed to the model’s ability to minimize uncertainty but also improved its precision in defect detection, a crucial aspect for real-time applications. This model architecture exhibits reduced susceptibility to overfitting and enhanced generalization.

The per-defect benefit of fusion can be interpreted physically: underextrusion produces both an intermittent AE burst (from discontinuous filament feed) and a small visible gap, neither of which is reliably detected alone—image-only modelsmiss gaps occluded by shadows, and AE-only models miss chronic low-amplitude flow drift. Warping is dominated by visible deformation but also produces low-frequency AE from bed-adhesion failure, again favoring the joint feature space. Pristine cases benefit from cross-modal agreement, which lowers the false-positive rate that drives unnecessary closed- loop interventions [29, 32].

The primary goal was to bridge critical gaps in manufacturing process monitoring and control, with an emphasis on real-time defect correction. By employing the DEF model for multimodal heterogeneous sensing, the study developed not only effective feedback control and remote monitoring capabilities but also laid the foundation for a digital thread. Furthermore, these findings represent a significant advancement in the application of digital thread technology in AM. The DEF model’s ability to incorporate the latest advances in deep learning architectures, coupled with its plug-and-play features, exemplifies an AI model that is versatile and adaptable. Additionally, the model’s proficiency in making rapid inferences from heterogeneous, multimodal sensor data is important in real-time manufacturing process monitoring. This capability is particularly relevant in the context of AM, where precision and speed are paramount. The study’s contributions thus extend beyond immediate process improvements, offering a broader understanding and utility of digital thread applications in the evolving landscape of AM.

The reported findings also highlight the impact of the DEF model on process monitoring and control in AM, significantly advancing digital thread technology. This development aligns with the goal of creating a more interconnected and responsive manufacturing environment. This integration is vital for maintaining product quality, as it facilitates real-time identification and correction of defects. Moreover, the DEF model’s capability to efficiently process multimodal sensor data underscores its potential to revolutionize industrial practices in AM. These advancements also have broader impacts on manufacturing efficiency, cost-effectiveness, and product quality. The improved monitoring and control enabled by data-driven models such as the DEF led to a reduction in defects and waste, resulting in significant cost savings and enhanced operational efficiency. In a domain where precision and reliability are crucial, consistent quality maintenance is essential. This change aligns with the objectives of digital thread technology, heralding a new standard for intelligent, interconnected manufacturing systems.

The extension of this work assesses the model’s robustness by evaluating its performance on layers that were not included in the training set. This assessment aimed to determine whether the model was overfitting to the training layer or capable of generalizing to multiple layers. The methodology used for this assessment involved utilizing unseen layers for validation. These layers were selected to span a range of conditions similar to those encountered in the operational environment. This approach was designed to test the model’s ability to adapt to new data that it had not been explicitly trained on, thereby providing a robust evaluation of its generalizability. To quantify the model’s performance on these new layers, metrics such as accuracy and recall were used. Accuracy measured the proportion of total correct predictions (both true positives and true negatives) among all predictions, while recall focused on the proportion of actual positives correctly identified by the model. The analysis phase focused on the rates of false positives and false negatives. The goal was to achieve high sensitivity (low false-negative rate) and high specificity (low false-positive rate). These metrics were crucial for understanding the model’s ability to accurately identify defects (sensitivity) while avoiding misclassifying nondefects as defects (specificity).

The intent of this study was to determine the model’s ability to generalize its learning to layers beyond those it was trained on. Table 6 shows that the test case for above the usual defect layer performed better than the one below the defect layer. The model was able to classify the defect layer (underextrusion) with 100% recall with the above test case, while the below test case had a recall of 0%. The model performs adequately with the above the usual layer, as indicated by the 87% accuracy.

Several failure modes can degrade DEF performance: (1) high ambient AE noise from co-located machinery can mask weak bursts, mitigated by adaptive noise-floor thresholding and the entropy-regularized loss [13]; (2) extreme lighting changes or reflective filaments can confound the optical branch, addressed by active illumination control and the modular replacement of the image feature extractor; (3) transparent or very dark filaments reduce optical contrast, in which case the AE branch carries more weight; (4) self-occluding geometries call for multi-camera setups; and (5) defect types absent from training (open-set) should be handled with uncertainty-aware outputs, an extension already prototyped in the authors’ remaining useful life (RUL) framework [13, 26].

Table 6. Robustness of the AM digital thread with respect to defect-layer location

This indicates that future work may involve changing the location of defect layers to below the usual layer if further robustness is required. Building on the findings of this study, future research should explore several key areas to further advance the application of digital thread technology in AM. Despite the contributions, this study acknowledges certain limitations in its research approach. The DEF model’s efficacy was tested under specific conditions, using a particular geometry and filament type. This specificity implies that if a different geometry or filament were used, the model would need to be retrained to maintain its accuracy. This underscores the need for additional development and testing to ensure the model’s robustness across diverse manufacturing conditions. One pivotal direction is expanding the DEF model’s capabilities to accommodate a broader range of geometries and filament types. Research should focus on enhancing the model’s adaptability and generalizability to ensure its effectiveness across various manufacturing scenarios. Source-systematic anomalies that produce a uniform extrusion drift across many layers were not explicitly seeded; per-layer features used by the DEF model remain informative for such cases, but a long-horizon prognostic model trained on continuous health indicators is a planned extension building on the authors’ prior RUL framework [13, 26]. The current dataset is sufficient for proof-of-concept but is unbalanced across defect classes (warping is represented by 3 datasets versus 18 for overextrusion and 11 for underextrusion) because of the longer bed-thermal-cycle time required to instigate warping; per-class balancing was enforced at the layer-frame level (50 defect/50 nondefect frames per test), but a broader collection of warping data is required before deployment.

Concluding Remarks

This research developed a multimodal deep learning framework for real-time integration of heterogeneous sensor data. Tested offline with custom loss functions and various defect scenarios, the model was then applied in an in situ monitoring testbed. Its distributed computing architecture across three layers prevented bottlenecks, enabling prompt defect correction. The framework’s automatic feature extraction and real- time feedback significantly improved confidence in defect detection for complex geometries. The main contribution of this work is the development of a digital thread framework for heterogeneous data streams to detect damage in real time and reach a decision-level state. One key development is the integration of a real-time DEF model with multimodal sensing. The model itself was unique in integrating two heterogeneous sensing streams for real-time classification. The model includes a custom loss function, which improved defect classification results. The DEF model has been designed to accommodate flexible data inputs, state-of-the-art model architectures, and outputs for various applications. This strategy facilitated the integration of various data types in real time, thereby enriching the depth and precision of defect detection. Moreover, the trained model was integrated with a real-time digital thread. Its contributions lay the foundation for future research, emphasizing the need for greater adaptability and the integration of advanced technologies in manufacturing.

Acknowledgments

The financial support that Sarah Malik received from the National Science Foundation (NSF) through the NSF Graduate Research Fellowship Program and the ASME Donald O. Thompson Graduate Research Fellowship, as well as the support she received from the Graduate Assistance in Areas of National Need (GAANN) fellowship, is greatly appreciated. Sincere gratitude is extended to RTX Corp. (formerly Raytheon Technologies) and MISTRAS Group for financial support and engagement in various parts of this study. Finally, special thanks to the members of the Theoretical and Applied Mechanics Group at Drexel University for their support, and especially to Hadi Khezam, Dhruv Shah, and Jason Voutsinas.

References

1. Wu, H., Y. Wang, and Z. Yu. 2016. “In situ monitoring of FDM machine condition via acoustic emission.” International Journal of Advanced Manufacturing Technology 84 (5-8): 1483–95. https://doi.org/10.1007/s00170-015-7809-4

2. Fu, Y., A. Downey, L. Yuan, A. Pratt, and Y. Balogun. 2021. “In situ moni toring for fused filament fabrication process: A review.” Additive Manufac turing 38: 101749. https://doi.org/10.1016/j.addma.2020.101749

3. Yoon, J., D. He, and B. Van Hecke. 2014. “A PHM approach to additive manufacturing equipment health monitoring, fault diagnosis, and quality control.” Annual Conference of the PHM Society 6 (1). https://doi.org/10.36001/phmconf.2014.v6i1.2338

4. Lu, Q. Y., and C. H. Wong. 2018. “Additive manufacturing process moni toring and control by non-destructive testing techniques: challenges and in-process monitoring.” Virtual and Physical Prototyping 13 (2): 39–48. https://doi.org/10.1080/17452759.2017.1351201

5. Ramírez, I. S., F. P. G. Márquez, and M. Papaelias. 2023. “Review on additive manufacturing and non-destructive testing.” Journal of Manufac turing Systems 66: 260–86. https://doi.org/10.1016/j.jmsy.2022.12.005

6. Ferraris, E., J. Zhang, and B. Van Hooreweder. 2019. “Thermography based in-process monitoring of Fused Filament Fabrication of polymeric parts.” CIRP Annals (68) 1: 213–216. https://doi.org/10.1016/j.cirp.2019.04.123

7. Tlegenov, Y., W. F. Lu, and G. S. Hong. 2019. “A dynamic model for current-based nozzle condition monitoring in fused deposition modelling.” Progress in Additive Manufacturing 4 (3): 211–23. https://doi.org/10.1007/s40964-019-00089-3

8. Rao, P. K., J. Liu, D. Roberson, Z. Kong, and C. Williams. 2015. “Online Real-Time Quality Monitoring in Additive Manufacturing Processes Using Heterogeneous Sensors.” Journal of Manufacturing Science and Engi neering 137 (6): 061007. https://doi.org/10.1115/1.4029823

9. Akhavan, J., and S. Manoochehri. 2022. “Sensory data fusion using machine learning methods for in-situ defect registration in additive manufacturing: a review.” In 2022 IEEE International IOT, Electronics, and Mechatronics Conference (IEMTRONICS), pp. 1-

10. IEEE. https://doi.org/10.1109/IEMTRONICS55184.2022.979581510.Wu,H.,Z.Yu,andY.Wang.2019.“Experimentalstudyoftheprocessfailurediagnosisinadditivemanufacturingbasedonacousticemission.”Measurement136:445–53.https://doi.org/10.1016/j.measurement.2018.12.067

11. Muktadir, M., M. N. Hasan, and M. Alam. 2022. “Additive manufac turing and acoustic Emission: A brief review.” Journal of Additive Manufac turing Technologies 2 (1): 632. https://doi.org/10.18416/JAMTECH.2212632

12. Wisner, B., K. Mazur, V. Perumal, K. P. Baxevanakis, L. An, G. Feng, and A. Kontsos. 2019. “Acoustic emission signal processing framework to identify fracture in aluminum alloys.” Engineering Fracture Mechanics. 210: 367–380. https://doi.org/10.1016/j.engfracmech.2018.04.027

13. Malik, S., and A. Kontsos. 2024. “A novel information entropy approach for crack monitoring leveraging nondestructive evaluation sensing.” Mechanical Systems and Signal Processing 214: 111207. https://doi.org/10.1016/j.ymssp.2024.111207

14. Liu, J., Y. Hu, B. Wu, and Y. Wang. 2018. “An improved fault diagnosis approach for FDM process with acoustic emission.” Journal of Manufac turing Processes 35: 570–79. https://doi.org/10.1016/j.jmapro.2018.08.038

15. Delli, U., and S. Chang. 2018. “Automated process monitoring in 3D printing using supervised machine learning.” Procedia Manufacturing 26: 865–70. https://doi.org/10.1016/j.promfg.2018.07.111

16. Gaja, H., and F. Liou. 2017. “Defects monitoring of laser metal deposition using acoustic emission sensor.” International Journal of Advanced Manufacturing Technology 90 (1-4): 561–74. https://doi.org/10.1007/s00170-016-9366-x

17. Wu, M., V. V. Phoha, Y. B. Moon, and A. K. Belman. 2016. “Detecting malicious defects in 3d printing process using machine learning and image classification.” Proceedings of the ASME 2016 International Mechanical Engineering Congress and Exposition (14): V014T07A004. https://doi.org/10.1115/IMECE2016-67641

18. Bacha, A., A. H. Sabry, and J. Benhra. 2019. “Fault Diagnosis in the Field of Additive Manufacturing (3D Printing) Using Bayesian Networks.” International Journal of Online & Biomedical Engineering 15 (3): 110–23. https://doi.org/10.3991/ijoe.v15i03.9375

19. Wang, C., X. Tan, S. Tor, and C. Lim. 2020. “Machine learning in additive manufacturing: State-of-the-art and perspectives.” Additive Manufacturing 36: 101538. https://doi.org/10.1016/j.addma.2020.101538

20. Meng, L., B. McWilliams, W. Jarosinski, H.-Y. Park, Y.-G. Jung, J. Lee, and J. Zhang. 2020. “Machine learning in additive manufacturing: A review.” Journal of the Minerals Metals & Materials Society 72 (6): 2363–77. https://doi.org/10.1007/s11837-020-04155-y

21. Menon, A., B. Póczos, A. W. Feinberg, and N. R. Washburn. 2019. “Optimization of silicone 3D printing with hierarchical machine learning.” 3D Printing and Additive Manufacturing 6 (4): 181–189. https://doi.org/10.1089/3dp.2018.0088

22. Mohammadi, M. G., D. Mahmoud, and M. Elbestawi. 2021. “On the application of machine learning for defect detection in L-PBF additive manufacturing.” Optics & Laser Technology 143: 107338. https://doi.org/10.1016/j.optlastec.2021.107338

23. Scime, L., and J. Beuth. 2018. “A multi-scale convolutional neural network for autonomous anomaly detection and classification in a laser powder bed fusion additive manufacturing process.” Additive Manufac turing 24: 273–86. https://doi.org/10.1016/j.addma.2018.09.034

24. Raza, A., A. Haider, and W. Haider. 2021. “A Consolidated Approach towards Application of Machine Learning Principles in Additive Manufac turing.” In 2021 IEEE International Conference on Electro Information Tech nology (EIT), pp. 1–6. IEEE. https://doi.org/10.1109/EIT51626.2021.9491833

25. Ogunsanya, M., J. Isichei, S. K. Parupelli, S. Desai, and Y. Cai. 2021. “In-situ droplet monitoring of inkjet 3D printing process using image analysis and machine learning models.” Procedia Manufacturing 53: 427–34. https://doi.org/10.1016/j.promfg.2021.06.045

26. Malik, S., and A. Kontsos. 2024. “Artificial intelligence for materials damage diagnostics and prognostics.” In Artificial Intelligence in Manufac turing, eds. Soroush, M. and R. D. Braatz, pp. 265–306. Academic Press. https://doi.org/10.1016/B978-0-323-99134-6.00004-9

27. Hall, D. L., and J. Llinas. 1997. “An introduction to multisensor data fusion.” Proceedings of the IEEE 85 (1): 6–23. https://doi.org/10.1109/5.554205

28. Liggins, M., II, D. Hall, and J. Llinas (eds.). 2017. Handbook of multisensor data fusion: theory and practice. CRC Press. https://doi.org/10.1201/9781420053098

29. Peng, X., L. Kong, W. Han, and S. Wang. 2022. “Multi-Sensor Image Fusion Method for Defect Detection in Powder Bed Fusion.” Sensors (Basel) 22 (20): 8023. https://doi.org/10.3390/s22208023

30. Rescsanski, S., M. Imani, and F. Imani. 2022. “Heterogeneous Sensing and Bayesian Optimization for Smart Calibration in Additive Manufacturing Process.” In ASME 2022 International Mechanical Engineering Congress and Exposition: V02BT02A051. https://doi.org/10.1115/IMECE2022-96010

31. Grasso, M., F. Gallina, and B. M. Colosimo. 2018. “Data fusion methods for statistical process monitoring and quality characterization in metal additive manufacturing.” Procedia CIRP 75: 103–07. https://doi.org/10.1016/j.procir.2018.04.045

32. Yang, Z., Y. Lu, S. Li, J. Li, Y. Ndiaye, H. Yang, and S. Krishnamurty. 2021. “In-process data fusion for process monitoring and control of metal additive manufacturing.” In ASME 2021 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference: V002T02A072. https://doi.org/10.1115/DETC2021-71813

33. Gaikwad, A., B. Giera, G. M. Guss, J.-B. Forien, M. J. Matthews, and P. Rao. 2020. “Heterogeneous sensing and scientific machine learning for quality assurance in laser powder bed fusion–A single-track study.” Additive Manufacturing 36: 101659. https://doi.org/10.1016/j.addma.2020.101659

34. Vandone, A., S. Baraldo, and A. Valente. 2018. “Multisensor data fusion for additive manufacturing process control.” IEEE Robotics and Automa tion Letters 3 (4): 3279–84. https://doi.org/10.1109/LRA.2018.2851792

35. Petrich, J., Z. Snow, D. Corbin, and E. W. Reutzel. 2021. “Multi-modal sensor fusion with machine learning for data-driven process monitoring for additive manufacturing.” Additive Manufacturing 48: 102364. https://doi.org/10.1016/j.addma.2021.102364

36. Wu, H., Z. Yu, and Y. Wang. 2016. “A new approach for online moni toring of additive manufacturing based on acoustic emission.” In ASME 2016 11th International Manufacturing Science and Engineering Conference: V003T08A013. https://doi.org/10.1115/MSEC2016-8551

37. Garanger, K., T. Khamvilai, and E. Feron. 2018. “3D printing of a leaf spring: A demonstration of closed-loop control in additive manufacturing.” 2018 IEEE Conference on Control Technology and Applications (CCTA), Copenhagen, Denmark, pp. 465–470. https://doi.org/10.1109/CCTA.2018.8511509

38. Reese, R., H. Bheda, and W. Mondesir. 2019. “Method to Monitor Additive Manufacturing Process for Detection and In-Situ Correction of Defects.” US Patent US10421267B2, issued 2019. https://patents.google.com/patent/US10421267B2/en

39. Wang, T., T.-H. Kwok, C. Zhou, and S. Vader. 2018. “In-situ droplet inspection and closed-loop control system using machine learning for liquid metal jet printing.” Journal of Manufacturing Systems 47: 83–92. https://doi.org/10.1016/j.jmsy.2018.04.003

40. Breese, P. P., T. Hauser, D. Regulin, S. Seebauer, and C. Rupprecht. 2021. “In situ measurement and closed-loop control for powder supply processes.” International Journal of Advanced Manufacturing Technology 116 (3–4): 889–903. https://doi.org/10.1007/s00170-021-07438-z

41. Rivera, F. J. M., and A. J. R. Arciniegas. 2020. “Additive manufacturing methods: Techniques, materials, and closed-loop control applications.” International Journal of Advanced Manufacturing Technology 109 (1–2): 17–31. https://doi.org/10.1007/s00170-020-05663-6

42. Malik, S., R. Rouf, K. Mazur, and A. Kontsos. 2020. “The industry Internet of Things (IIoT) as a methodology for autonomous diagnostics in aerospace structural health monitoring.” Aerospace (Basel, Switzerland) 7 (5): 64. https://doi.org/10.3390/aerospace7050064

43. Malik, S., R. Rouf, K. Mazur, and A. Kontsos. 2020. “A Dynamic Data Driven Applications Systems (DDDAS)-Based Digital Twin IoT Frame work.” In Dynamic Data Driven Applications Systems, eds. Darema, F., E. Blasch, S. Ravela, and A. Aved, pp. 29–36. DDDAS 2020, Boston, MA. Springer Cham. https://doi.org/10.1007/978-3-030-61725-7_6

44. Malik, S., K. Mazur, R. Rouf, and A. Kontsos. 2019. “The industry internet of things (IIoT) as a methodology for autonomous diagnostics, prognostics in aerospace structural health monitoring.” Structural Health Monitoring 2019. https://doi.org/10.12783/shm2019/32214

45. Menéndez, M., J. Pardo, L. Pardo, and M. Pardo. 1997. “The jensen- shannon divergence.” Journal of the Franklin Institute 334 (2): 307–18. https://doi.org/10.1016/S0016-0032(96)00063-4

46. Malik, S., and A. Kontsos. 2022. “An Information Theory Approach for Internet of Things Enabled Damage Monitoring.” In ASME 2022 Confer ence on Smart Materials, Adaptive Structures and Intelligent Systems: V001T05A008. https://doi.org/10.1115/SMASIS2022-91119

47. Malik, S., and A. Kontsos. 2023. “Implementation of Information Entropy in an Industrial Internet of Things Approach for Structural Health Monitoring Applications.” Structural Health Monitoring 2023. https://doi.org/10.12783/shm2023/36907

48. Baldi, P., and P. J. Sadowski. 2013. Understanding dropout.” Advances in Neural Information Processing Systems 26: 2814–2822. https://proceedings.neurips.cc/paper/2013/hash/71f6278d140af599e06ad9bf1ba03cb0-Abstract.html

49. Eren, L., T. Ince, and S. Kiranyaz. 2019. “A generic intelligent bearing fault diagnosis system using compact adaptive 1D CNN classifier.” Journal of Signal Processing Systems for Signal, Image, and Video Technology 91 (2): 179–89. https://doi.org/10.1007/s11265-018-1378-3

50. He, J., X. Li, Y. Chen, D. Chen, J. Guo, and Y. Zhou. 2021. “Deep transfer learning method based on 1D-CNN for bearing fault diagnosis.” Shock and Vibration 2021: 6687331. https://doi.org/10.1155/2021/6687331

51. Ciaburro, G., and G. Iannace. 2022. “Machine-learning-based methods for acoustic emission testing: A review.” Applied Sciences (Basel, Switzer land) 12 (20): 10476. https://doi.org/10.3390/app122010476

52. Essien, A., and C. Giannetti. 2020. “A deep learning model for smart manufacturing using convolutional LSTM neural network autoencoders.” IEEE Transactions on Industrial Informatics 16 (9): 6069–78. https://doi.org/10.1109/TII.2020.2967556

53. Imoto, K., T. Nakai, T. Ike, K. Haruki, and Y. Sato. 2019. “A CNN-based transfer learning method for defect classification in semiconductor manu facturing.” IEEE Transactions on Semiconductor Manufacturing 32 (4): 455–59. https://doi.org/10.1109/TSM.2019.2941752

54. Wang, Y., K. Hong, J. Zou, T. Peng, and H. Yang. 2020. “A CNN-based visual sorting system with cloud-edge computing for flexible manu facturing systems.” IEEE Transactions on Industrial Informatics 16 (7): 4726–35. https://doi.org/10.1109/TII.2019.2947539

55. Feng, X., X. Gao, and L. Luo. 2021. “A ResNet50-based method for clas sifying surface defects in hot-rolled strip steel.” Mathematics 9 (19): 2359. https://doi.org/10.3390/math9192359

56. Wen, L., X. Li, and L. Gao. 2020. “A transfer convolutional neural network for fault diagnosis based on ResNet-50.” Neural Computing & Applications 32 (10): 6111–24. https://doi.org/10.1007/s00521-019-04097-w

57. Hu, Z., Z. Yin, L. Qin, and F. Xu. 2022. “A novel method of fault diagnosis for injection molding systems based on improved Vgg16 and machine vision.” Sustainability (Basel) 14 (21): 14280. https://doi.org/10.3390/su142114280

58. Ledesma, R., and A. Ramlatchan. 2021. “Convolutional Neural Networks for Image Classification in Metal Selective Laser Melting Additive Manufacturing.” Paper presented at the Conference on Computer Vision and Pattern Recognition (CVPR 2021), 21–24 June, Nashville, TN. https://ntrs.nasa.gov/citations/20205009555

59. Fakih, M. A. H., S. K. Singh, S. Mustapha, M. Radzienski, and P. H. Malinowski. 2023. “A comprehensive analysis of guided-wave propagation in 3D-printed PLA plates with different infill densities – Experimental study.” Mechanical Systems and Signal Processing 202: 110645. https://doi.org/10.1016/j.ymssp.2023.110645

60. Fakih, M. A., S. K. Singh, S. Mustapha, and P. Malinowski. 2022. “Damage Localization in 3D-Printed Plates with Different Infill Densities.” In Proceedings of the ASME 2022 49th Annual Review of Progress in Quantitative Nondestructive Evaluation, pp. 52–58. https://doi.org/10.1115/QNDE2022-95348