The Department of Defense (DOD) is seeking innovative solutions through the Small Business Innovation Research (SBIR) program for the development of tools aimed at evaluating AI/ML models on incompletely-labeled satellite imagery datasets. The primary objective is to create a flexible system that can reliably estimate the performance of AI/ML models in the presence of label noise, which is a common issue in overhead imagery datasets, thereby enhancing confidence in model performance post-deployment. This opportunity is particularly significant for organizations like the National Geospatial-Intelligence Agency (NGA) that require robust evaluation methods for various computer vision tasks, including object detection and automated target recognition. Interested parties should note that the solicitation is currently in the pre-release phase, with the open date set for April 23, 2025, and a close date of May 21, 2025, by which applications must be submitted. For further details, applicants can visit the DOD SBIR website at https://www.dodsbirsttr.mil/topics-app/.

## Description

** All work will be conducted on classified data at the TS/SCI level.**Deployment and maintenance of AI/ML models at scale requires frequent, reliable test and evaluation. The trustworthiness of such evaluation depends on two factors: well-defined performance metrics and well-defined test data. Performance metrics are extensively studied, as are techniques to split datasets into training and test partitions, so the emphasis of this topic is the development of tools to support the use of such test partitions.A key, unresolved issue in the use of test data is the presence of label noise, such as missing or incomplete annotations, poorly sized, oriented or aligned localizations (bounding boxes), or mislabelled annotation names (misclassifications). Such noise is accepted to be present in most AI/ML datasets, and it has been determined to have potentially significant impacts on evaluation quality for a range of tasks and fields [1, 2], but few methods exist to mitigate these impacts. Tools to identify and correct label noise are available, but require significant human effort for large-scale data, making them insufficient for most applications. Forgoing a labeled test set entirely and estimating performance based on training data or unlabeled data may also be viable [3, 4, 5], but this has not been extensively studied, particularly in the domain of overhead imagery where errors may be highly correlated with environmental conditions (for example, difficulties in annotating cloudy imagery).A flexible system to reliably estimate true AI/ML model performance without a need for perfect test data would fill a significant gap in capability for organizations working with large numbers of models and large amounts of data and (such as NGA) as well as organizations without the time or human capital to exhaustively assess datasets used for AI/ML development (such as academic labs and smaller commercial entities). Such a system would greatly enhance organizations&rsquo; confidence in post-deployment model performance without incurring additional cost and delay.The primary interest of this topic is in tools for evaluation of object detection and automated target recognition tasks, but other computer vision tasks such as image classification, semantic segmentation, change detection, etc. are within scope.

## Objective

Develop tooling for rapid performance analysis of diverse AI/ML models on overhead imagery datasets with label noise, over wide ranges of environmental conditions.

## Phase I

Demonstrate a basic software package for the estimation of AI/ML model performance on computer vison detector tasks for overhead satellite imagery. Methods should be robust to noise or absence of test data. Prototype metrics to quantify the accuracy of the developed methodology (that is, the gap between observed performance on imperfect data and the actual performance of the model). Methods should generalize to multiple real overhead imagery datasets over multiple architectures. Deliver all software packages and a summary report of performance investigations.

## Phase II

Apply methods and tooling of Phase I to overhead imagery. Provide more detailed analyses of performance, such evaluations for various combinations of collection geometries and environmental conditions. Develop techniques to extrapolate performance estimates to novel data which may be unlabeled, partially labeled, and/or labeled under a different ontology. Software from Phase II should be compatible with multiple overhead imagery datasets over multiple model architectures. Demonstrate tooling and methodologies on novel detector development without preexisting large datasets on at least three challenge problems developed in consort with NGA sponsor. Deliver all software packages and a summary report of performance investigations. Performers are expected to accomplish this work at the TS/SCI level.

## Phase III: Dual Use Applications

Enhance software from Phase II to function with minimal human intervention on large-scale datasets and AI/ML models in line with commercial applications. Extensively document usage and functionality of all developed software. Deliver all software packages and a summary report of performance investigations.

Topic

Agency

Program

Topic Link

Additional Information