Generating Higher Resolution Sky Maps Using a Deep Gaussian Process Poisson Model

Selection of Initial Points Using Latin Hypercube Sampling for Active Learning

Roelof Coetzer, North-West University

Binary classification is a common task in machine learning where the goal is to assign binary labels to observed and new observations. However, labelling large sets of data is often a time-consuming and expensive process. Active learning learns from a few data points, while selecting the most informative unlabelled samples for labelling to improve model performance. The success of active learning is dependent on the selection of the initial points to initialize the active learning process, and the selection criteria used to identify informative samples. In this paper we illustrate the use of Latin Hypercube sampling, conditioned Latin Hypercube sampling, and a modified Latin Hypercube sampling procedure for initializing active learning for the estimation of the logistic regression classifier. In addition to the usual performance measures for classification, we consider the mean squared error of the predicted posterior probability and the classifier as performance measures. The results are demonstrated using simulated data sets and some actual case studies. We show that Latin Hypercube sampling outperforms the traditional random sampling approach for various performance measures.

Optimal Experimental Designs for Process Robustness Studies

Peter Goos, KU Leuven

In process robustness studies, experimenters are interested in comparing the responses at different locations within the normal operating ranges of the process parameters to the response at the target operating condition. Small differences in the responses imply that the manufacturing process is not affected by the expected fluctuations in the process parameters, indicating its robustness. In this presentation, we propose an optimal design criterion, named the generalized integrated variance for differences (GID) criterion, to set up experiments for robustness studies. GID-optimal designs have broad applications, particularly in pharmaceutical product development and manufacturing. We show that GID-optimal designs have better predictive performances than other commonly used designs for robustness studies, especially when the target operating condition is not located at the center of the experimental region. In some situations that we encountered, the alternative designs typically used are roughly only 50% as efficient as GID-optimal designs. We will demonstrate the advantages of tailor-made GID-optimal designs through an application to a manufacturing process robustness study of the Rotarix liquid vaccine.

 

October 10th

Session 4 (8-9:30AM)

BayesFLo: Bayesian Fault Localization of Complex Software Systems

Irene Ji, Duke

Software testing is essential for the reliable development of complex software systems. A key step in software testing is fault localization, which uses test data to pinpoint failure-inducing combinations for further diagnosis. Existing fault localization methods, however, are largely deterministic, and thus do not provide a principled approach for assessing probabilistic risk of potential root causes, or for integrating domain and/or structural knowledge from test engineers. To address this, we propose a novel Bayesian fault localization framework called BayesFLo, which leverages a flexible Bayesian model on potential root cause combinations. A key feature of BayesFLo is its integration of the principles of combination hierarchy and heredity, which capture the structured nature of failure-inducing combinations. A critical challenge, however, is the sheer number of potential root cause scenarios to consider, which renders the computation of posterior root cause probabilities infeasible even for small software systems. We thus develop new algorithms for efficient computation of such probabilities, leveraging recent tools from integer programming and graph representations. We then demonstrate the effectiveness of BayesFLo over state-of-the-art fault localization methods, in a suite of numerical experiments and in two motivating case studies on the JMP’s XGBoost interface.

On Forming Control Limits for Short Run Standardized Xbar Control Charts with Varying Subgroup Sizes

Annie Dudley and Di Michelson, JMP and Bill Woodall, Virginia Tech

Short Run control charts are commonly used when manufacturing more than one product on the same production line. Developments with High Throughput and Just In Time Manufacturing methods lead to multiple products made on the same line, together with shorter intervals between products. Short Run control charts enable the practitioner to view all products in the order they were made all on the same chart, by either centering all the data by a corresponding product mean or target, or by standardizing all the data by corresponding product target and estimate of sigma.
In the absence of Short Run control chart methods, a practitioner cannot either view all the data on a meaningful control chart or apply any of the Nelson Runs Tests, as the runs are not in order.
Current methods used for computing control limits for Short Run, Standardized and Summarized (XBar/R) Control Charts have deficiencies when the subgroup size is not constant. In practice, XBar/R charts might regularly have unequal subgroup sizes. In this paper, we explore methods to resolve the constant subgroup size constraint.
We consider different published sets of recommendations for control limit calculation of Short Run standardized XBar/R control charts. For each, we study the distributional foundations of the estimators and run a simulation study to verify chart performance and present our findings for both the estimate of sigma and control limits.

A Novel Robust MCUSUM for Phase II Profile Monitoring

Abdel-Salam G. Abdel-Salam, Qatar University

Previous studies in Phase II analysis, as applied to real-life scenarios, predominantly focus on profile monitoring, assuming that Phase I model estimations and control limit settings are accurately executed without any model misspecifications. However, this assumption may not always hold true, potentially leading to imperfect fittings between the response and independent variables. Addressing this gap, our research introduces two novel, robust Multivariate CUSUM control charts employing non-parametric and semi-parametric methodologies for Phase II profile monitoring using linear mixed models. These advancements aim to efficiently detect shifts of varying magnitudes in the slope parameter, accommodating a range of profiles, sample sizes, and levels of model misspecification across both in-control and out-of-control scenarios, applicable to uncorrelated and correlated profiles alike.
We extensively evaluated the proposed control charts against traditional parametric methods through thorough simulation studies and a real-world application, employing Average Run Length (ARL) and Extra Quadratic Loss (EQL) as benchmark criteria. Our findings reveal that the semi-parametric based multivariate CUSUM control chart exhibits superior performance and heightened sensitivity in detecting shifts across the spectrum, surpassing both parametric and non-parametric approaches.

Nonparametric Online Monitoring of Dynamic Networks

Peihua Qiu, University of Florida

Network sequence has been commonly used for describing the longitudinal pattern of a dynamic system. Proper online monitoring of a network sequence is thus important for detecting temporal structural changes of the system. To this end, there have been some discussions in the statistical process control (SPC) literature to first extract some features from the observed networks and then apply an SPC chart to monitor the extracted features sequentially over time. However, the features used in many existing methods are insensitive to some important network structural changes, and the control charts used cannot accommodate the complex structure of the extracted features properly. In this paper, we suggest using four specific features to describe the structure of an observed network, and their combination can reflect most network structural changes that we are interested in detecting in various applications. After the four features are extracted from the observed networks, we suggest using a multivariate nonparametric control chart to monitor the extracted features online. Numerical studies show that our proposed network monitoring method is more reliable and effective than some representative existing methods in various cases considered.

A Graphical Comparison of Screening Designs using Support Recovery Probabilities

Kade Young, Eli Lilly & Co.

A screening experiment attempts to identify a subset of important effects using a relatively small number of experimental runs. Given the limited run size and a large number of possible effects, penalized regression is a popular tool used to analyze screening designs. In particular, an automated implementation of the Gauss-Dantzig selector has been widely recommended to compare screening design construction methods. Here, we illustrate potential reproducibility issues that arise when comparing screening designs via simulation, and recommend a graphical method, based on screening probabilities, which compares designs by evaluating them along the penalized regression solution path. This method can be implemented using simulation, or, in the case of lasso, by using exact local lasso sign recovery probabilities. Our approach circumvents the need to specify tuning parameters associated with regularization methods, leading to more reliable design comparisons.

Session 5 (10-11:30AM)

Machine Learning, Cross Validation, and DOE

Maria Weese, Miami University

Recently there is interest in fitting complex models to traditional experimental designs (i.e. machine learning models and central composite designs). Machine learning models, which require out of sample data for hyperparameter tuning, are often optimized with cross validation which involves selecting subsets of training and test data. When larger, less structured, data sets are used to train machine learning models there is little concern with the structure of the subsets since they are not likely to be different from the full data. However, when the training data is collected using a structured experimental design, creating subsets using cross validation might produce training samples that do not preserve the structure of the original design. In this work we investigate the consequences of using cross validation on data collected using various types of experimental designs. We provide a literature review that illustrates the types of models fit to designed experiments and how those models are tuned. Finally, we present designs constructed using an optimality criteria to mitigate the effects of sampling in leave-one-out cross validation and k-fold cross validation procedures.

Autonomy versus Safety: Joint Modeling of Disengagement and Collision Events in Autonomous Vehicle Driving Study

Simin Zheng, Virginia Tech

As the popularity of artificial intelligence (AI) continues to grow, AI systems have become increasingly embedded into various aspects of daily life, leading to significant transformations in industries and changing how people live. One of the typical applications of AI systems is autonomous vehicles (AVs). In AVs, the relationship between the level of autonomy and safety is an important research question to answer, which can lead to two types of recurrent events data being recorded: disengagement and collision events. This paper proposes a joint modeling approach with multivariate random effects to analyze these two types of recurrent events data. The proposed model captures the inter-correlation between the levels of autonomy and safety in AVs. We apply an expectation-maximization (EM) algorithm to obtain maximum likelihood estimates for the functional form of fixed effects, variance-covariance components, and baseline intensity functions. This proposed joint modeling approach can be useful for modeling recurrent events data with multiple event types from various applications. We analyze disengagement and collision events data from the California Department of Motor Vehicles AV testing program to demonstrate its application.

A Replacement for Lenth’s Method for Nonorthogonal Designs

Caleb King, JMP

A key part of implementing a screening experiment is the correct identification of the active factors. To do this requires separating the signal from the noise. Therefore, a precise estimate of σ is desirable. Lenth (1989) provided a method for estimating σ in the context of unreplicated factorial designs that has become the standard for factor screening. Currently, optimal designs are in common use as screening designs as they exist for any desired number of runs. The price for using these designs is often the loss of orthogonality of the effects. For nonorthogonal designs then, a replacement for Lenth’s method is necessary. This work provides a method for analyzing saturated nonorthogonal screening designs. We show that our method performs well even when the number of active effects is up to half the number of runs.

Optimal Two-level Designs Under Model Uncertainty

Steven Gilmour, King’s College London

Two-level designs are widely used for screening experiments where the goal is to identify a few active factors which have major effects. We apply the model-robust Q_B criterion for the selection of optimal two-level designs without the usual requirements of level balance and pairwise orthogonality. We provide a coordinate exchange algorithm for the construction of Q_B-optimal designs for the first-order maximal model and second-order maximal model and demonstrate that different designs will be recommended under different prior beliefs. Additionally, we study the relationship between this new criterion and the aberration-type criteria. Some new classes of model-robust designs which respect experimenters’ prior beliefs are found.

Monitoring Univariate Processes Using Control Charts: Some Practical Issues and Advice

Bill Woodall, Virginia Tech

We provide an overview and discussion of some issues and guidelines related to monitoring univariate processes with control charts. We offer some advice to practitioners to help them set up control charts appropriately and use them most effectively. We propose a four-phase framework for control chart set-up, implementation, use, and maintenance. In addition, our recommendations may be useful for researchers in the field of statistical process monitoring. We identify some current best practices, some misconceptions, and some practical issues that rely on practitioner judgment.

How Generative AI models such as ChatGPT can be (Mis)Used in SPC Practice, Education, and Research? An Exploratory Study

Fadel Megahed, Miami University

Generative Artificial Intelligence (AI) models such as OpenAI’s ChatGPT have the potential to revolutionize Statistical Process Control (SPC) practice, learning, and research. However, these tools are in the early stages of development and can be easily misused or misunderstood. In this paper, we give an overview of the development of Generative AI. Specifically, we explore ChatGPT’s ability to provide code, explain basic concepts, and create knowledge related to SPC practice, learning, and research. By investigating responses to structured prompts, we highlight the benefits and limitations of the results. Our study indicates that the current version of ChatGPT performs well for structured tasks, such as translating code from one language to another and explaining well-known concepts but struggles with more nuanced tasks, such as explaining less widely known terms and creating code from scratch. We find that using new AI tools may help practitioners, educators, and researchers to be more efficient and productive. However, in their current stages of development, some results are misleading and wrong. Overall, the use of generative AI models in SPC must be properly validated and used in conjunction with other methods to ensure accurate results.

Session 6 (1:30-3PM)

A Causal Discovery Inspired Approach for Improving Interpretation of Multivariate T² Control Charts.

Justin Greene, Rutgers

One challenge when monitoring a multivariate process using Hotelling’s T² is identifying which variable, or set of variables, is causing an out-of-control signal. Decomposing T² into independent terms has been shown to effectively identify the variable(s) of concern. However, such decomposition, in addition to being computationally challenging, does not naturally render itself to a causal analysis of out-of-control signals. One promising approach to reduce the computational load and improve the root-cause investigation, is to assume a causal directed acyclic graph (DAG) associated with the charting variables. However, the existing framework assumes that the DAG is known apriori based on domain knowledge, which is unlikely in most real-life situations. In this paper, we explore methods based on causal discovery for inferring the DAG based on historical or phase-I data and using estimated DAGS to improve root-cause analysis of out-of-control signals. 

Quantitative Assessment of Machine Learning Reliability and Resilience

Lance Fiondella, Dartmouth

Advances in machine learning (ML) have led to applications in safety-critical domains, including security, defense, and healthcare. These ML models are confronted with dynamically changing and actively hostile conditions characteristic of real-world applications, requiring systems incorporating ML to be reliable and resilient. Many studies propose techniques to improve the robustness of ML algorithms. However, fewer consider quantitative techniques to assess the reliability and resilience of these systems. To address this gap, this study demonstrates how to collect relevant data during the training and testing of ML suitable for the application of software reliability, with and without covariates, and resilience models and the subsequent interpretation of these analyses. The proposed approach promotes quantitative risk assessment of machine learning technologies, providing the ability to track and predict degradation and improvement in the ML model performance and assisting ML and system engineers with an objective approach to compare the relative effectiveness of alternative training and testing methods. The approach is illustrated in the context of an image recognition model, which is subjected to two generative adversarial attacks and then iteratively retrained to improve the system’s performance. Our results indicate that software reliability models incorporating covariates characterized the misclassification discovery process more accurately than models without covariates. Moreover, the resilience model based on multiple linear regression incorporating interactions between covariates tracks and predicts degradation and recovery of performance best. Thus, software reliability and resilience models offer rigorous quantitative assurance methods for ML-enabled systems and processes.

Quick Input-Response Space-Filling (QIRSF) Designs

Xiankui Yang, University of South Florida

Space-filling designs have been broadly used in computer experiments to guide efficient and informative data collection. Traditional space-filling designs primarily focus on uniformly spreading design points throughout the input space. Recent development on input-response space-filing (IRSF) designs offer additional advantages when having a nice coverage over the range of response values is also desirable for some applications. The original IRSF designs use the maximin distance criterion and a modified point exchange algorithm to balance the uniform spread of design points across the input space and response values. In this paper, we develop a new quick input-response space-filling (QIRSF) design which utilizes hierarchical clustering techniques and the minimax point to achieve desirable coverage in input and response spaces. The new method dramatically reduces the computing time by at least 20 folds while maintaining high efficiency of approximating the IRSF designs. The performance and computational efficiency of the proposed methods are demonstrated through multiple examples with different input and response dimensions and varied characteristics of response surfaces. An R Shiny App is offered to facilitate easy construction of QIRSF designs of flexible size and dimension.

A Kernel-Based Approach for Modelling Gaussian Processes with Functional Information
Andrew Brown, Clemson

Gaussian processes are commonly used tools for modeling continuous processes in machine learning and statistics. This is partly due to the fact that one may employ a Gaussian process as an interpolator for a finite set of known points, which can then be used for prediction and straight forward uncertainty quantification at other locations. However, it is not always the case that the available information is in the form of a finite collection of points. For example, boundary value problems contain information on the boundary of a domain, which is an uncountable collection of points that cannot be incorporated into typical Gaussian process techniques. In this paper, we propose and construct Gaussian processes that unify, via reproducing kernel Hilbert space, the typical finite case with the case of having uncountable information by exploiting the equivalence of conditional expectation and orthogonal projections. We show existence of the proposed Gaussian process and that it is the limit of a conventional Gaussian process conditioned on an increasing but finite number of points. We illustrate the applicability via numerical examples and proof-of-concept.

Optimal Experimental Designs for Precision Medicine with Multi-component Treatments

Yeng Saanchi, NC State

In recent years, many medical and behavioral interventions have evolved to combine multiple therapeutic options. We refer to treatments that consist of a combination of therapeutic options as multi-component treatments. Cancer treatment is one area in which the use of such treatments is increasingly becoming common. Considering several therapeutic options in a clinical trial can result in a potentially large number of unique treatment combinations at each decision point. Ignoring the overlap among components and viewing each combination as unique in evaluating the treatment combinations is inefficient. We propose a strategy that leverages classical optimal design methods to assign combination treatments to maximize power for primary and secondary analyses of interest. Our method uses the c and D optimality criteria in constructing a composite objective function to be minimized to obtain the optimal allocation of multi-component treatments to promote the patient’s long-term well-being.

Simulation Experiment Design for Calibration via Active Learning

Ozge Surer, Miami University

Simulation models often have parameters as input and return outputs to understand the behavior of complex systems. Calibration is the process of estimating the values of the parameters in a simulation model in light of observed data from the system that is being simulated. When simulation models are expensive, emulators are built with simulation data as a computationally efficient approximation of an expensive model. An emulator then can be used to predict model outputs, instead of repeatedly running an expensive simulation model during the calibration process. Sequential design with an intelligent selection criterion can guide the process of collecting simulation data to build an emulator, making the calibration process more efficient and effective. This presentation focuses on two novel criteria for sequentially acquiring new simulation data in an active learning setting by considering uncertainties on the posterior density of parameters. Analysis of several simulation experiments and real-data simulation experiments from epidemiology demonstrates that proposed approaches result in improved posterior and field predictions.