Test Design and Model Validation for DoD Weapon Systems

Overview

The webinar, part of the DataWorks workshop, presents three examples of applying statistical methods to Department of Defense testing: adaptive operational testing of the KC-46, sequential test design for ballistic limits, and validation of a torpedo simulation. Across the talks, recurring themes include testing earlier, designing tests efficiently, connecting test results to operational decisions, and building confidence in models before relying on them.

1. Adaptive Relevant Testing for the KC-46

Quinn Stank describes Adaptive Relevant Testing (ART) as AFOTEC’s approach to delivering useful capability sooner than traditional testing. Traditional approaches often wait until a system is fully ready before conducting a lengthy evaluation and publishing results.

Developmental and operational testing

• Developmental testing focuses on whether a system meets technical requirements.
• Operational testing asks whether crews can accomplish real missions with the system in realistic conditions.
• Stank argues that both perspectives are needed and should be integrated.

Applying ART to acquisition trade-offs

ART applies to the trade-offs among schedule, performance, and cost. Its principles include:

• Involve operational testers early and move testing earlier in the schedule.
• Test systems when they are ready rather than waiting for every component or certification.
• Tailor and adapt test plans to the situation while maintaining sound statistical design.
• Combine developmental and operational testing where practical.
• Use continuing feedback and communicate findings in time for decision-makers to act.

Designing tests around KC-46 missions

The KC-46 is intended for aerial refueling, airlift, and aeromedical evacuation, with additional command-and-control and situational-awareness capabilities.

The test program includes five critical operational issues: three focused on effectiveness and two on suitability.

• Effectiveness testing covers mission performance, including refueling, cargo and passenger operations, and aeromedical evacuation.
• Suitability testing considers whether the aircraft can be generated to meet mission needs and whether it can be supported logistically.

The team used design-of-experiments methods, collecting about 4,500 data points across roughly 700 test events.

Testing incrementally as capabilities become available

At the start of operational testing, only three of the 17 desired receiver aircraft were FAA-certified for testing. Rather than wait for all 17, the team began with the available aircraft and added testing as certifications arrived.

About 22% of test points were completed before the official start of operational testing. This approach helped identify issues sooner and allowed testing on other capabilities to continue when one system was unavailable.

Examples of early findings

• Cargo-floor locks: During early testing, a test crew found that cargo locks could become unlocked in flight. The team reported the deficiency, and a design change—a button that holds the latch in place—was developed. Testing continued on other capabilities while cargo operations were paused for safety.
• Takeoff and landing data (TOLD): Boeing’s dispatchers could provide TOLD for test flights, but operational crews would need the aircraft to calculate it itself. Operational assessment identified this as a major deficiency. The aircraft later calculated its own data using the aircraft performance tool.

Limitations and lessons

A firm fixed-price contract limited access to some contractor test data and made changes harder. Other challenges included:

• Some systems may not be ready when early testing begins, leading to inefficient scheduling or repeated discovery of issues.
• Early testing increases demands on personnel and can make it harder to distinguish a system fault from a training or crew issue.

Stank’s main lessons are to integrate developmental and operational testing, test what is available, account for contract constraints, and communicate results promptly at a level decision-makers can use.

2. D-Optimal Sequential Testing for Ballistic Limits

Lenny Lombardo presents a sequential test method for ballistic-limit testing. In these tests, a projectile is fired at armor at different velocities, and the probability of penetration is modeled as a function of velocity.

Why a more complex model may be needed

In a simple case, penetration probability increases with velocity. However, a change in the projectile–armor interaction can make the response non-monotonic. For example, a projectile may remain intact at low velocity, penetrate at intermediate velocity, and shatter at higher velocity.

The talk uses a Chang-Bod model to represent this kind of “change in mechanism.”

Motivation for a more efficient test design

Existing procedures may use 80 shots over a lower velocity range, followed by another 60 shots under a modified approach, for a total of 140 shots.

The proposed approach uses D-optimal design, which selects test velocities to efficiently estimate parameters of interest, including the 50% penetration velocity and the response-curve slope.

How the sequential method works

• When the mechanism for each shot is known, the problem can be simplified into three generalized linear models.
• The initial design aims to create regions with mixed outcomes—partials and completes—so the model parameters can be estimated.
• After each stage, the method chooses the next velocity by maximizing the determinant of the information matrix.
• Knowing the mechanism generally improves the design and analysis. When the mechanism is unknown, optimization is more difficult and can encounter convergence problems. The study used a cubic-fit fallback when convergence failed.

Simulation findings and next steps

The simulations examined different amounts of separation between mechanisms and sample sizes from 40 to 100.

• Estimates of the location parameters showed little median bias.
• Slope estimates showed small-sample bias, which diminished as sample size increased.
• Greater overlap between mechanisms makes it harder to estimate mechanism-specific parameters, especially when the mechanism is unknown.

Future work includes developing likelihood-based confidence intervals for cases where the mechanism is known.

3. Validation of the Environment-Centric Weapons Analysis Facility

Elliot Bartis discusses validation of the Environment-Centric Weapons Analysis Facility (ECWAF), a hardware-in-the-loop torpedo simulation. Its purpose is to determine whether the simulation’s predictions are trustworthy enough to support testing and operational decisions.

Why simulation validation matters

Live torpedo tests are expensive and constrained by safety, logistics, and the availability of submarines. A simulation can explore conditions that are difficult or impossible to test live, including potential operating environments.

Bartis emphasizes that simulation cannot entirely replace live tests; it can help target live testing where it is most valuable.

What ECWAF models

The facility connects torpedo hardware to a computer simulation of the environment, targets, and torpedo movement and sonar.

Environmental conditions matter: shallow or complex environments can produce acoustic reflections and other signals that challenge the torpedo more than the open ocean. ECWAF was upgraded to be environment-centric so it could represent those effects more realistically.

Validation approach

1. Run the simulation under designed conditions and collect outcomes such as hits or misses.
2. Build an emulator—a statistical model of the simulation’s predictions, using methods such as logistic regression.
3. Apply the emulator to conditions from real, in-water tests and compare its predictions with observed outcomes.

The team used a two-phase process: first rehearsing with historical data and an older torpedo configuration, then moving to the current system with newly designed in-water tests.

The intended uses are staged from easier to harder. The most ambitious goal is to use the simulation to predict torpedo effectiveness in operational test reporting.

Assessing validation

The team evaluated accuracy, specificity (correctly predicting misses), sensitivity (correctly predicting hits), and the Brier score, which measures the quality of probabilistic predictions.

In the first phase:

• Accuracy was 71%.
• Specificity was 32%.
• Sensitivity was 83%.
• The median Brier score was 0.07.

The results suggested that the simulation tended to overpredict performance, but the errors were not concentrated in a particular scenario. This supported proceeding to the next phase while improving the test design.

Factors supporting the validation effort

• Sustained Navy and Department of Defense investment in the simulation.
• Early involvement of DOT&E and IDA in accreditation planning.
• A substantial archive of live torpedo-test data.
• Frequent working-level meetings and open communication among stakeholders.
• Torpedo and simulation developers working side by side, reducing integration problems.
• Clearly specified intended uses for the model.

Potential payoff

Bartis estimates that a live test costs about $200,000, compared with roughly $200 for a simulation run. If validation succeeds, the program expects to reduce live firings by more than half for a future torpedo variant, while retaining live testing for information the simulation cannot provide.

The validation model is intended to evolve as it incorporates more historical and future test data.

Speakers and Sources Featured

• Rebecca Medlin — Institute for Defense Analyses (IDA) research staff member and workshop organizer; moderator.
• Quinn Stank — First lieutenant and lead analyst for the KC-46 at AFOTEC Detachment 5; presented on Adaptive Relevant Testing.
• Lenny Lombardo — Analyst at the U.S. Army Aberdeen Test Center; presented on D-optimal sequential ballistic-limit testing.
• Elliot Bartis — Research staff member at IDA; presented on validation of the Environment-Centric Weapons Analysis Facility.
• Audience participants — Asked questions during the webinar Q&A; most were not identified in the subtitles.