This report provides the U.S. Department of the Air Force (DAF) with a guide for strengthening its process for reviewing the performance of its Systems Engineering and Integration federally funded research and development centers. The guide is intended to help DAF establish the purpose and scope of its review process, design a system for grading performance, engage with stakeholders, and follow up on performance reviews.

Getting More of What You Need

A Guide for Developing a Stronger Performance Review Process for Systems Engineering Support

Victoria A. Greenfield, Laura Werber, Colby P. Steiner, Sandra Kay Evans, Lisa Pelled Colabella, James D. Powers

ResearchPublished Jul 9, 2024

Cover: Getting More of What You Need
Download PDF
Additional Downloads

Includes Research Summary

This report provides the U.S. Department of the Air Force (DAF) with a guide for strengthening its process for reviewing the performance of its Systems Engineering and Integration (SE&I) federally funded research and development centers (FFRDCs). The guide is intended to help DAF establish the purpose and scope of its review process, design a system for grading the performance of its SE&I FFRDCs, collect and analyze evidence on their performance, engage with stakeholders, and follow up on performance reviews with remediation or recognition and ongoing evaluations of the process.

In the report, the research team walks through the main elements of a review process, including conceptualization, design, implementation, and follow-up, with consideration of benefits and costs. Although the guide was developed for the U.S. Air Force Missile Systems Center (SMC), which is now Space Systems Command, it should also be of interest to other federal agencies that seek to strengthen or replace a process for reviewing systems engineering or similar support.

Key Findings

  • A rigorous performance review process starts with conceptualization, which means establishing the vision for the process, consisting of (1) the purpose, which is the “why” or reason for the process, and (2) the scope, which is the “who, what, where, and when” of the process, broadly stated.
  • Steps, roles, and responsibilities should be set out early in the design phase, before an agency stands up and implements the process.
  • The grading system is the structured set of tools that an agency uses to assess—review and rate, or grade—the performance of its support. A complete system should include grading areas, grading factors, and performance measures that address an agency's organizational priorities, goals and concerns, and standards for assigning grades.
  • An agency must decide what kinds of evidence it needs in order to gauge whether its SE&I FFRDCs are meeting, exceeding, or falling short of expectations. If an FFRDC is exceeding or falling short, the agency must determine in what ways and by how much to either recognize the accomplishment or address the deficit.
  • Research on change management and program evaluation suggests that an effective review process requires acceptance and support from stakeholders, including FFRDC leaders, and that training and transparency matter, both on their own merit and to facilitate buy-in.
  • A process, once created, should not be treated as static. The process should be reappraised continually—during and after reviews with its own evaluation—and substantial changes should be tested and validated.

Topics

Document Details

Citation

Chicago Manual of Style

Greenfield, Victoria A., Laura Werber, Colby P. Steiner, Sandra Kay Evans, Lisa Pelled Colabella, and James D. Powers, Getting More of What You Need: A Guide for Developing a Stronger Performance Review Process for Systems Engineering Support. Santa Monica, CA: RAND Corporation, 2024. https://www.rand.org/pubs/research_reports/RRA737-3.html.
BibTeX RIS

Research conducted by

This research was prepared for the Department of the Air Force and conducted within the Resource Management Program of RAND Project AIR FORCE.

This publication is part of the RAND research report series. Research reports present research findings and objective analysis that address the challenges facing the public and private sectors. All RAND research reports undergo rigorous peer review to ensure high standards for research quality and objectivity.

This document and trademark(s) contained herein are protected by law. This representation of RAND intellectual property is provided for noncommercial use only. Unauthorized posting of this publication online is prohibited; linking directly to this product page is encouraged. Permission is required from RAND to reproduce, or reuse in another form, any of its research documents for commercial purposes. For information on reprint and reuse permissions, please visit www.rand.org/pubs/permissions.

RAND is a nonprofit institution that helps improve policy and decisionmaking through research and analysis. RAND's publications do not necessarily reflect the opinions of its research clients and sponsors.