The Federal Emergency Management Agency’s (FEMA’s) Qualification System (FQS) is a credentialing process that focuses on standardizing personnel abilities through training and applied experience. FEMA’s Integration Branch asked the authors of this report to identify existing gaps and areas for improvement and examine ways to measure and monitor the effectiveness of FQS in the future.

Identifying Gaps and Areas for Improvement in the FEMA Qualification System for Incident Workforce Positions

Recommendations for Developing an Improvement and Evaluation Process

Leslie Adrienne Payne, Andrea M. Abler, Susan G. Straus, Jason Michel Etchegaray

ResearchPublished Mar 6, 2024

To fulfill its mission objectives, the Federal Emergency Management Agency (FEMA) requires a well-trained workforce. The Post-Katrina Emergency Management Reform Act of 2006 mandated that FEMA develop a personnel credentialing process for incident workforce personnel responding to disaster situations. FEMA, in turn, created the FEMA Qualification System (FQS), a credentialing process that focuses on standardizing personnel abilities through training and applied experience.

Following concerns raised about the efficiency and effectiveness of FQS to provide the needed structure to qualify individuals for their assigned positions, FEMA's Integration Branch asked the authors to identify existing FQS gaps and areas for improvement and examine ways to measure and monitor the effectiveness of FQS in the future. This report details the authors’ approach to addressing both issues and the related findings so that FEMA can ensure the development of employees who provide vital disaster recovery services to the United States and its people.

Key Findings

  • Overall, FQS has a fair number of strengths, and stakeholders see value in the concept behind FQS and its component parts.
  • In practice, the component pieces do not always function as designed.
  • There are very few systematic measures and evaluation activities to gauge whether processes in FQS components are implemented effectively and in ways that align with FEMA's goals.

Recommendations

  • Develop a standardized checklist consisting of required Position Task Book elements and a checklist that documents when tasks from Position Task Books and content from training are aligned.
  • Analyze C&E ratings for common rating errors and for possible differences in ratings by protected class or other factors.
  • Institute supervisor oversight and conduct performance review of C&Es.
  • If none currently exists, establish a grievance/redress process for trainees who feel unfairly evaluated by C&Es.
  • For Kirkpatrick Level 2 evaluations, create pre-tests and post-tests to assess improvement in learning.
  • For Level 2 evaluations, analyze the psychometric properties of tests, ideally using item response theory.
  • Consider using a quantitative model that combines numeric ratings on inputs currently used by the Qualification Review Board to make qualification recommendations.
  • Consider an alternate Qualification Review Board model that better addresses the technical nuances of each cadre and job position. For example, this could take the form of convening, for each cadre, a technical review panel with specialists specific to that cadre.
  • Determine the frequency with which users select Incident Workforce personnel for deployment by name rather than relying on the Deployment Tracking System algorithm, and conduct a qualitative analysis of justifications.

Topics

Document Details

Citation

Chicago Manual of Style

Payne, Leslie Adrienne, Andrea M. Abler, Susan G. Straus, and Jason Michel Etchegaray, Identifying Gaps and Areas for Improvement in the FEMA Qualification System for Incident Workforce Positions: Recommendations for Developing an Improvement and Evaluation Process. Santa Monica, CA: RAND Corporation, 2024. https://www.rand.org/pubs/research_reports/RRA2460-1.html.
BibTeX RIS

Research conducted by

This research was sponsored by FEMA's Integration Branch and conducted in the Disaster Management and Resilience Program of the RAND Homeland Security Research Division.

This publication is part of the RAND research report series. Research reports present research findings and objective analysis that address the challenges facing the public and private sectors. All RAND research reports undergo rigorous peer review to ensure high standards for research quality and objectivity.

RAND is a nonprofit institution that helps improve policy and decisionmaking through research and analysis. RAND's publications do not necessarily reflect the opinions of its research clients and sponsors.