- What techniques can be employed to assess program execution performance and risks as early as possible?
- How can program managers and stakeholders leverage available government and external data to assess these concerns?
- What data sources are available and useful?
- What practical concerns arise when implementing such assessments?
Getting early indication of potential contractor performance risks and contract execution issues is critical for proactive acquisition management. When contractors are in danger of not meeting contractual performance goals, Department of the Air Force (DAF) acquisition management may not be fully aware of the shortfall until, for example, a schedule deadline is missed, government testing indicates poor system’s technical performance, or costs exceed expectations.
Concerns continue to be raised about cost and schedule growth in acquisition and experts postulate about a lack of knowledge about the status of acquisition programs. In this report, the authors focus on metrics to identify emerging execution problems earlier than traditional acquisition oversight systems to enable more-proactive risk and performance management. They summarize their findings, which include a taxonomy of contractor relative risks, leading indicators of performance, relevant data sources, risk measures and equations, and a prototype that implements some of these findings using real data sources. This research should be of interest to acquisition professionals and leadership who are searching for ways to improve acquisition performance through early identification of potential relative contractor risks and execution problems to inform active program management and mitigation of risks. The prototype should be of interest to acquisition officials (from program managers to milestone decision authorities) to help them access more data in an easy-to-understand way so they can focus their limited time on areas that require increased management attention. This approach should be useful during any phase of the acquisition process.
- Automated tools can be created to ingest, aggregate, and analyze data that could focus managers’ limited resources on early indicators of performance issues and potential risk indicators buried in large, diverse data; this could help inform mitigating actions by management based on effectiveness, program relevance, and risk tolerance.
- Some data that are important for assessing relative contractor risks are very difficult to obtain—even for DAF officials and federally funded research and development centers.
- Despite the limitations of this research prototype, this approach is more sophisticated in some ways (e.g., through company-level metrics, such as financial health or supply chain risks and predictive indicators of future performance) than other available systems and might point to features or concepts that could be added to DAF or U.S. Department of Defense systems that assess potential contractor risks.
- A taxonomy of potential risk measures beyond those traditionally examined in program management and that use relative performance against their peers or fixed thresholds could highlight risk indicator outliers to government acquisition professionals.
- The integration and assessment of traditional and nontraditional data sources could provide useful indications of potential areas of concern.
- With further refinement, this approach could be a powerful tool to help program managers and stakeholders leverage vast amounts of data to identify performance and risk indicators for further due diligence, confirmation, and proactive management.
- Additional data and the inclusion and validation of more metrics and implementation details are needed to make this prototype more robust; data availability, accessibility, and analysis are key.
- Officials should develop a research prototype database and software system to test and refine the concept. They should also cross-reference data sources to associate contracts and contractors with DAF programs and build a simple user interface to view results from both a contractor and program perspective.
Table of Contents
The Challenge: Early Identification of Contractor Execution Problems in Acquisition
Indicators of Performance
Prototype Structure and Approach
Insights, Conclusions, and Next Steps
Sentiment Modeling and Analysis and Modeling of Free-Text Assessments and Associated Ratings
Assessing Final Cost and Schedule Using a Rayleigh Function