Skip to content
Home » Blog » Summary of Findings of the Evaluation of the CSA 2010 Operational Model Test

Summary of Findings of the Evaluation of the CSA 2010 Operational Model Test


The U.S. Department of Transportation’s Federal Motor Carrier Safety Administration
(FMCSA) released the University of Michigan Transportation Research Institute (UMTRI)
independent evaluation of the Compliance, Safety, Accountability (CSA) program’s
Operational Model Test (Op-Model Test) on August 31, 2011. UMTRI’s findings confirm
that CSA substantially improves FMCSA’s enforcement and compliance model. The results
confirm that the CSA model enables FMCSA and its State Partners to contact more
commercial motor carriers earlier to correct safety problems and ensure compliance
with safety regulations in order to reduce crashes, injuries, and fatalities related
to commercial motor vehicles.

Launched in 2008, the CSA Op-Model Test divided motor carriers from four test states
(Colorado, Georgia, Missouri, and New Jersey) between test and control groups. UMTRI
evaluated the effectiveness of the new Safety Measurement System (SMS) and CSA interventions,
and compared the cost and efficiency of the CSA compliance and enforcement model
to the previous model. They found effectiveness and efficiency gains that fully
support the ongoing national implementation of CSA, as outlined below. FMCSA added
additional states, Delaware, Kansas, Maryland, Minnesota, and Montana, to the test
to demonstrate full implementation challenges and to provide a validation dataset
for evaluation purposes.

CSA’s SMS better identifies motor carriers for safety interventions than the previous
SafeStat system.

  • “The results showed that the SMS is a significant improvement over the SafeStat
    system in identifying unsafe carriers. (p. xiv)”
  • Crash rates were higher for motor carriers identified with safety problems in the
    SMS’s seven Behavior Analysis and Safety Improvement Categories (BASICs) than for
    motor carriers that were not identified with safety problems in the seven BASICs.
  • The crash rate for motor carriers that were identified with safety problems by the
    SMS in the Unsafe Driving BASIC was more than three times greater than the crash
    rate for motor carriers not identified with any safety problems by SMS.

CSA interventions are effective in improving motor carriers’ safety behavior.

  • “The effect of the warning letter intervention is likely one of the most significant
    findings in this evaluation. (p. xviii)” Twelve months after receiving a warning
    letter, SMS results showed that 83% of test carriers had resolved identified safety
    problems and only 17% continued to have safety problems.
  • The new CSA Onsite Focused Investigations proved to be effective. Almost 20% fewer
    motor carriers continued to show safety problems 12 months after an on-site focused
    investigation, as compared with those receiving traditional Compliance Reviews (CRs).

CSA interventions use enforcement resources efficiently.

  • More intensive interventions were used on carriers that exhibited higher crash risk
    confirming that the rules guiding intervention selection are operating to ensure
    effective and efficient safety interventions.
  • Warning letters, which were found to be very effective in improving safety behavior,
    had only a nominal cost.
  • CSA Onsite Focused Investigations cost approximately 53 percent less than CRs and
    were effective in producing compliance.
  • The average cost of CSA interventions was $754 per motor carrier, as compared to
    $1438 for motor carriers receiving CRs.

CSA reaches more carriers to improve safety compliance.

  • CSA interventions contact approximately three times the number of motor carriers
    contacted using the previous model which relied primarily on CRs.
  • Among the CSA test group, the annual percentage of motor carriers contacted was
    9.9 percent, compared with the 3.2 percent of motor carriers that received full
    CRs in 2009.

The evaluator identified some areas that require improvement and FMCSA is firmly
committed to a continuous improvement process for this very important program.

SMS’s BASICs are significantly related to underlying motor carrier safety, although
the Cargo-Related and Driver Fitness BASICs show a weaker relationship to crash
risk.

  • The evaluator’s findings are in line with FMCSA’s effectiveness findings; as a result,
    at the end of the Op-Model Test FMCSA adjusted how it identifies motor carriers
    for intervention to ensure BASICs with the strongest relationship to future crashes
    receive the most emphasis. However, FMCSA continues to address motor carriers with
    patterns of noncompliance in the Cargo-Related and Driver Fitness BASICs, which
    include carrier requirements for being properly licensed, carrying medical cards
    to allow verification that a driver meets the medical qualification standards, adequately
    securing cargo, and properly packaging and handling hazardous materials.
  • As part of its ongoing commitment to continually assess and improve the SMS, FMCSA
    has a study underway that may result in improvements to some BASICs, with particular
    effect on the Cargo-Related BASIC.

There was lag time in measureable safety performance improvement after CSA investigations,
and for carriers with the most serious safety problems, improvement rates were similar
to those of the control group.

  • FMCSA expects the upcoming Safety Fitness Determination rulemaking to accelerate
    return to compliance or removal from service for motor carriers with the worst safety
    problems.
  • Based on lessons learned in the Op-Model Test, FMCSA improved the CSA investigative
    process and training in the Safety Management Cycle for its Federal and State Partner
    investigators. The enhanced investigative process allows investigators to systematically
    identify motor carriers’ safety problems and to recommend remedies to help carriers
    to quickly become safer.

View the full
UMTRI Report
(PDF, 6.3 MB).