A Family of Orthogonal Main Effects Screening Designs for Mixed Level Factors

Predictive Resilience Modeling

Priscila Silva and Lance Fiondella, University of Massachusetts Dartmouth

Resilience is the ability of a system to respond, absorb, adapt, and recover from a disruptive event. Dozens of metrics to quantify resilience have been proposed in the literature. However, fewer studies have proposed models to predict these metrics or the time at which a system will be restored to its nominal performance level after experiencing degradation. This talk presents alternative approaches to model and predict performance and resilience metrics with elementary techniques from reliability engineering and statistics. We will also present a free and open source tool developed to apply the models without requiring detailed understanding of the underlying mathematics, enabling users to focus on resilience assessments in their day to day work.

Semi-destructive Gage R&R: De-confounding Variance Components in Systems with Predictable Degradation

Douglas Gorman, BD

Sometimes the process of measuring a part changes the characteristic being measured due to deformation, wear, or other degradation mechanism. In these situations, it is natural to treat the test method as destructive, since repeat measurements exhibit variation that is not pure repeatability, but the combination of repeatability and the change in performance of the item being measured. Common strategies for performing gage RR studies with destructive test methods confound part-to-part variation with measurement system repeatability. In cases where part-to-part variation is large, destructive tests often fail typical Gage R&R acceptance criteria. When measured parts exhibit a predictable degradation effect with repeat measurements, repeat measures can de-confound the part-to-part variation and repeatability error. Including the degradation effect in the gage R&R ANOVA model enables the part-to-part variation to be independently estimated. This talk explains how to design an analyze a gage study in such a semi-destructive situation and provides examples from industry.

 

October 6th

Session 4 (8-9:30AM)

Monitoring Time-Varying Networks with Principal Component Network Representations

James Wilson, University of San Francisco

We consider the problem of network representation learning (NRL) for network samples and how NRL can be used for monitoring change in a sequence of time-varying networks. We first introduce a technique, Principal Component Analysis for Networks (PCAN), that identifies statistically meaningful low-dimensional representations of a network sample via subgraph count statistics. We then describe a computationally fast sampling-based procedure, sPCAN, that not only is significantly more efficient than its counterpart, but also enjoys the same advantages of interpretability. We show how the features from (s)PCAN can be used to detect local changes in a sequence of time-varying networks. We explore the time series of co-voting networks of the US Senate paying close attention to eras of political polarization as well as harmonization. If time permits, we will discuss large-sample properties of the methods when the sample of networks analyzed is a collection of kernel-based random graphs.

County-Level Surveillance of COVID Counts

Steven E. Rigdon, Saint Louis University

We fit spatially correlated models at the county level for weekly COVID data. Poor data quality in many regions requires special attention. By applying a conditional (spatial) autoregressive model, we are able to assess the effect of a number of variables, such as demographics, vaccine rates, etc., on the transmission rate of the disease.

Empirical Calibration for a Nonparametric Lower Tolerance Bound

Caleb King, JMP

In many industries, the reliability of a product is often determined by a quantile of a distribution of a product’s characteristics meeting a specified requirement. A typical approach to address this is to assume a distribution model and compute a one-sided confidence bound on the quantile. However, this can become difficult if the sample size is too small to reliably estimate a parametric model. Linear interpolation between order statistics is a viable nonparametric alternative if the sample size is sufficiently large. In most cases, linear extrapolation from the extreme order statistics can be used, but can result in inconsistent coverage. In this talk, we will present an empirical study used to generate calibrated weights for linear extrapolation that greatly improves the accuracy of the coverage across a feasible range of distribution families with positive support. We will demonstrate this calibration technique using two examples from industry.

Constructing Control Charts for Autocorrelated Data Using an Exhaustive Systematic Samples Pooled Variance Estimator

Scott Grimshaw, Brigham Young University

SPC with positive autocorrelation is well known to result in frequent false alarms if the autocorrelation is ignored. The autocorrelation is a nuisance and not a feature that merits modeling and understanding. This paper proposes exhaustive systematic sampling, which is similar to Bayesian thinning except no observations are dropped, to create a pooled variance estimator that can be used in Shewhart control charts with competitive performance. The expected value and variance are derived using quadratic forms that is nonparametric in the sense no distribution or time series model is assumed. Practical guidance for choosing the systematic sampling interval is offered to choose a value large enough to be approximately unbiased and not too big to inflate variance. The proposed control charts are compared to time series residual control charts in a simulation study that validates using the empirical reference distribution control limits to preserve stated in-control false alarm probability and demonstrates similar performance.

Augmenting Definitive Screening Designs: Going Outside the box

Robert Mee, University of Tennessee Knoxville

Definitive screening designs (DSDs) have grown rapidly in popularity since their introduction by Jones and Nachtsheim (2011). Their appeal is that the second-order response surface (RS) model can be estimated in any subset of three factors, without having to perform a follow-up experiment. However, their usefulness as a one-step RS modeling strategy depends heavily on the sparsity of second-order effects and the dominance of first-order terms over pure quadratic terms. To address these limitations, we show how viewing a projection of the design region as spherical and augmenting the DSD with axial points in factors found to involve second-order effects remedies the deficiencies of a stand-alone DSD. We show that augmentation with a second design consisting of axial points is often the Ds-optimal augmentation, as well as minimizing the average prediction variance. Supplemented by this strategy, DSDs are highly effective initial screening designs that support estimation of the second-order RS model in three or four factors.

Session 5 (10-11:30AM)

PERCEPT: a New Online Change-point Detection Method Using Topological Data Analysis

Simon Mak, Duke University

Topological data analysis (TDA) provides a set of data analysis tools for extracting embedded topological structures from complex high-dimensional datasets. In recent years, TDA has been a rapidly growing field which has found success in a wide range of applications, including signal processing, neuroscience and network analysis. In these applications, the online detection of changes is of crucial importance, but this can be highly challenging since such changes often occur in low-dimensional embeddings within high-dimensional data streams. We thus propose a new method, called PERsistence diagram-based ChangE-PoinT detection (PERCEPT), which leverages the learned topological structure from TDA to sequentially detect changes. PERCEPT follows two key steps: it first learns the embedded topology as a point cloud via persistence diagrams, then applies a non-parametric monitoring approach for detecting changes in the resulting point cloud distributions. This yields a non-parametric, topology-aware framework which can efficiently detect online geometric changes. We investigate the effectiveness of PERCEPT over existing methods in a suite of numerical experiments where the data streams have an embedded topological structure. We then demonstrate the usefulness of PERCEPT in two applications on solar flare monitoring and human gesture detection.

Statistical Approaches for Testing and Evaluating Foundation Models

Giri Gopalan and Emily Casleton, Los Alamos National Laboratory, Olivier Binette, Duke University, and Rachel Longjohn, University of California Irvine

A modern thrust of research in artificial intelligence (AI) has focused on foundation models (FMs) – deep neural networks, typically involving a transformer architecture and trained on a prodigious corpus of data in a self-supervised fashion, that can be adapted to solve downstream tasks. The introduction of FMs has been referred to as a paradigm shift away from “narrow AI”, where models are built and trained for specific tasks and generally do not perform well on out-of-distribution data. Before these models are to be trusted by domain scientists, a crucial aspect is to rigorously evaluate FM performance and quantify uncertainties when testing, evaluating, and comparing different FMs, especially as FMs (and derivatives thereof) become widely used in a variety of critical applications; a specific example is for event detection and characterization with an FM model built on seismic data. Given the fact that FMs are so large (GPT3, the engine behind the popular ChatGPT, contains 175 billion parameters) and that they are able to address multiple tasks, traditional approaches used to evaluate and quantify uncertainty of AI models need to be recast for FMs.

We will discuss statistically grounded ways to perform uncertainty quantification with FMs, focusing on salient topics such as comparing and contrasting extrinsic and intrinsic evaluations of FMs, adapting to situations where resampling of the training data and iteratively re-fitting FMs is not computationally feasible, and combining information from performance on different tasks for an assessment of FM performance. For instance, an FM that achieves a low loss score when being trained in a self-supervised manner might perform poorly on a set of tasks related to a particular application context – hence, appropriately combining extrinsic and intrinsic FM performance depends on the specific use case. For another example, a battery of tasks that the FM is tested on might have substantial variance in quality, difficulty, and number of questions, all of which should be considered when combining task-specific performance metrics. Finally, it is imperative that comparisons of FM performance account for uncertainty on the metric so that superior performance cannot be attributed to sampling variability, for example. We will attempt to suggest best practices for dealing with issues such as those aforementioned, and our work should be helpful to practitioners who wish to evaluate and assess FMs from a statistically informed perspective.

Practical Design Strategies for Robust Nonlinear Statistical Modelling

Timothy E. O’Brien, Loyola University of Chicago

Researchers often find that generalized linear or nonlinear regression models are applicable for scientific assessment including physical, agricultural, environmental and synergy modelling. In line with other processes these nonlinear models perform better than linear models since they tend to fit the data well and the associated model parameter(s) are typically scientifically meaningful. Generalized nonlinear model parameter estimation presents statistical modelling challenges including computational, convergence and curvature issues. Further, researchers are also often in a position of requiring optimal or near-optimal (so-called “robust”) designs for the given chosen nonlinear model. A common shortcoming of most optimal designs for nonlinear models used in practical settings, however, is that these designs typically focus only on (first-order) parameter variance or predicted variance, and thus ignore the inherent nonlinear of the assumed model function. Another shortcoming of optimal designs is that they often have only p support points, where p is the number of model parameters. This talk examines modelling and estimation methods connected with generalized nonlinear models useful for physical, environmental and biological modelling and provides concrete novel suggestions related to robust design strategies for these models. Numerous examples will be provided and software methods will be discussed.

Entropy-Driven Design of Sensitivity Experiments

David M. Steinberg, Tel Aviv University, Rotem Rozenblum, Consultant, and Amit Teller, Rafael Advanced Defense Systems Ltd.

Sensitivity experiments assess the relationship of a stress variable to a binary response, for example how drop height affects the probability of product damage. These experiments usually employ a sequential design where the next stress level is chosen based on results from the data obtained thus far. For example, the popular Bruceton (up/down) experiment increases the stimulus if no response is observed and decreases it following a positive response. In this work we present a new entropy-driven design algorithm that chooses stress levels to maximize the mutual information between the next observation and the goals of the inference. The approach is parametric, so the goals could be the parameters themselves or functions of them, such as a quantile or a set of quantiles. We use a Bayesian analysis that quantifies the entropy of the goals from the current posterior distribution. The method can also be used when there are two or more stress variables (e.g. drop height and the hardness of the landing surface). We illustrate the use of the algorithm by simulation and on real experiment.

Problem Framing: Essential to Successful Statistical Engineering Applications

Roger Hoerl, Union College

The first two phases of the statistical engineering process are to identify the problem, and to properly structure it. These steps relate to work that is often referred to elsewhere as framing of the problem. While these are obviously critical steps, we have found that problem-solving teams often “underwhelm” these phases, perhaps being over-anxious to get to the analytics. This approach typically leads to projects that are “dead on arrival” because different parties have different understandings of what problem they are actually trying to solve. In this expository article, we point out evidence for a consistent and perplexing lack of emphasis on these first two phases in practice, review some highlights of previous research on the problem, offer tangible advice for teams on how to properly frame problems to maximize the probability for success, and share some real examples of framing challenging problems.

Testing the Prediction Profiler with Disallowed Combinations—A Statistical Engineering Case Study

Yeng Saanchi, North Carolina State University

The prediction profiler is an interactive display in JMP statistical software that allows a user to explore the relationships between multiple factors and responses. A common use case of the profiler is for exploring the predicted model from a designed experiment. For experiments with a constrained design region defined by disallowed combinations, the profiler was recently enhanced to obey such constraints. In this case study, we show how our validation approach for this enhancement touched upon many of the fundamental principles of statistical engineering. The team approached the task as a design of experiments problem.

Session 6 (1:30-3PM)

Multi-criteria Evaluation and Selection of Experimental Designs from a Catalog

Mohammed Saif Ismail Hameed, José Núñez Ares, and Peter Goos, University of Leuven

In recent years, several researchers have published catalogs of experimental plans. First, there are several catalogs of orthogonal arrays, which allow experimenting with two-level factors as well as multi-level factors. The catalogs of orthogonal arrays with two-level factors include alternatives to the well-known Plackett-Burman designs. Second, recently, a catalog of orthogonal minimally aliased response surface designs (or OMARS designs) appeared. OMARS designs bridge the gap between the small definitive screening designs and the large central composite designs, and they are economical designs for response surface modeling. Third, catalogs of D- and A-optimal main-effect plans have been enumerated. Each of these catalogs contains dozens, thousands or millions of experimental designs, depending on the number of runs and the number of factors, and choosing the best design for a particular problem is not a trivial matter. In this presentation, we introduce a multi-objective method based on graphical tools to select a design. Our method analyzes the trade-offs between the different experimental quality criteria and the design size, using techniques from multi-objective optimization. Our procedure presents an advantage compared to the optimal design methodology, which usually considers only one criterion for generating an experimental design. Additionally, we will show how our methodology can be used for both screening and optimization experimental design problems. Finally, we will demonstrate a novel software solution, illustrating its application for a few industrial experiments. 

Experiments in Time and Space

Mindy Hotchkiss, Aerojet Rocketdyne

Design of Experiments (DOE) provides a structured approach to test planning that can provide important insights to users into the interrelationships between factors under consideration. Much material is available on the design and analysis of studies, but there is limited guidance on incorporating more complex factors into experimental structures, such as those with temporal and/or spatial dependencies. Time- and space-related factors can introduce unexpected uncertainties; if unrecognized or ignored, these may negatively impact the quality and repeatability of an experiment. More than just constraints or limitations, these factors can also be leveraged for greater experimental effectiveness. This talk discusses the practical aspects of DOE implementation with respect to incorporating temporal and spatial considerations into designed experiment structures during the experimental planning phase. Topics discussed include the multidimensional nature of measurement, specimen design, and spatial arrangement in 2 and 3 dimensions.

Multi-layer Sliced Designs with Application in AI Assurance

Qing Guo, Virginia Tech, Peter Chien, University of Wisconsin, Madison, and Xinwei Deng,

Motivated by the importance of enhancing AI assurance with respect to configuration and hyper-parameters in AI algorithms, this work provides an experimental design angle to address this challenging problem. Our focus is to conduct an efficient experimental design to quantify and detect the effects of hyper-parameters affecting the performance of AI algorithms. Specifically, we propose a multi-layer sliced design to enable quantifying the effects of sliced factors and design factors, such that it can account for hyper-parameters having different effects under different configurations of the AI algorithm. Moreover, we develop a novel estimation procedure to estimate the effects of these factors and test for their significance. The performance of the proposed design and analysis is evaluated by both simulation studies and real-world AI applications.

Planning Reliability Assurance Tests for Autonomous Vehicles
Simin Zheng and Yili Hong, Virginia Tech, Lu Lu, University of South Florida, and Jian Liu, University of Arizona

Artificial intelligence (AI) technology has become increasingly prevalent and transforms our everyday life. One important application of AI technology is the development of autonomous vehicles (AV). AV combines robotic process automation and AI, which has become an essential representation of AI applications. Hence, the reliability of an AV needs to be carefully demonstrated via an assurance test so that the product can be used with confidence in the field. To plan for an assurance test, one needs to determine how many AVs need to be tested for how many miles. Existing research has made great efforts in investigating reliability demonstration tests in the other fields of study for product development and assessment. However, statistical methods have not been utilized in AV test planning. This paper aims to fill in this gap by developing statistical methods for planning AV reliability assurance tests based on recurrent event data. We explore the relationship between multiple criteria of interest in the context of planning reliability assurance tests. Specifically, two test plans based on homogeneous and non-homogeneous Poisson processes are developed with recommendations offered for practical uses. The disengagement events data from the California Department of Motor Vehicles AV testing program is used to illustrate proposed assurance testing methods. We conclude the paper with suggestions for practitioners

Assessing Variable Activity for Bayesian Regression Trees

Akira Horiguchi, Duke University and Matthew T. Pratola and Thomas J. Santner, The Ohio State University

Bayesian Additive Regression Trees (BART) are non-parametric models that can capture complex exogenous variable effects. In any regression problem, it is often of interest to learn which variables are most active. Variable activity in BART is usually measured by counting the number of times a tree splits for each variable. Such one-way counts have the advantage of fast computations. Despite their convenience, one-way counts have several issues. They are statistically unjustified, cannot distinguish between main effects and interaction effects, and become inflated when measuring interaction effects. An alternative method well-established in the literature is Sobol´ indices, a variance-based global sensitivity analysis technique. However, these indices often require Monte Carlo integration, which can be computationally expensive. This paper provides analytic expressions for Sobol´ indices for BART posterior samples. These expressions are easy to interpret and computationally feasible. We also provide theoretical guarantees of contraction rates. Furthermore, we will show a fascinating connection between first-order (main-effects) Sobol´ indices and one-way counts. Finally, we compare these methods using analytic test functions and the En-ROADS climate impacts simulator.

The Effectiveness and Flexibility of a Predictive Distribution Approach to Process Optimization with Multiple Responses

John Peterson, PDQ Research and Consulting and Enrique del Castillo, The Pennsylvania State University

One can often view a manufacturing process as a stochastic process that has several sources of variation and uncertainty. There is variation due to input materials, ambient conditions, operator-to-operator changes, etc. In addition, the use of predictive models involves parameter uncertainty and model form lack-of-fit. Therefore, process optimization in the face of such uncertainties, and complicated by the presence of multiple quality responses, can be a challenge for effective and flexible quantification and associated statistical inference. Over the past two decades several papers have appeared which have utilized predictive distribution approaches to effectively address multiple response process optimization problems. These include both early and later stage optimization problems. This presentation will provide an overview of the predictive distribution approach and discuss computational tools as well.