When children bring home a report card from school, part of their grades come from how often they made it to class or turned in homework. But the larger part comes from how they did on tests, in class and on take-home assignments. In other words, how much they've learned or how hard they're trying.

MORE FROM THE LAB: Subscribe to our weekly newsletter

These days, several organizations give hospitals report cards, combining measurements of how well hospitals serve their patients and keep them safe. Some use only objective patient outcomes, such as infection rates, while others allow hospitals to report on their efforts to improve safety.

But a new study finds that a well-known hospital grading system may put too much weight on the wrong things.

The study, published in the journal Medical Care by a University of Michigan team, takes an in-depth look at the Hospital Safety Grade system, run by The Leapfrog Group, a nonprofit organization that focuses on patient safety.

The researchers find that most hospitals that participate in Leapfrog's Hospital Survey self-report that they're in perfect compliance with most, if not all, of the Safe Practices in the survey. But researchers show that how a hospital did on these measures had little in common with independent measurements of hospital-acquired infections, or with whether the government penalized the hospital for high infection or readmission rates.

"The Safe Practices part of the Hospital Safety Grade is based on whether a hospital self-reports that it has adopted certain protocols, and not whether those protocols are actually followed," says Jennifer Meddings, M.D., M.Sc., the study's senior author and an assistant professor of internal medicine at the U-M Medical School.

For example, she says, one Safe Practices measure involves hand hygiene.

"It may be news to consumers that this measure involves hospital self-reports of having specific hand-washing protocols in place, not how often health care workers are washing their hands."

She also notes that any one hospital's Safe Practices score counts toward its overall Hospital Safety Grade based on a comparison with the average from all participating hospitals. So hospitals receive little grade advantage by reporting perfect scores — but are at risk of much lower grades if they report less-than-perfect scores.

The researchers note that Leapfrog will issue a new grading formula next month, but not in time to calculate the new grades that will come out soon after.

To self-report or not to self-report?

Hospital Safety Grades combine data from national sources, including public data from the Centers for Medicare and Medicaid Services and the Centers for Disease Control and Prevention — and, for hospitals that choose to participate, the voluntary Leapfrog Hospital Survey.

SEE ALSO: How Big Data Brings Big Gains in Surgical Quality

Just more than half of hospitals in the study completed the Leapfrog survey, which includes self-reporting of eight National Quality Forum Safe Practices measures. If a hospital reports them, they account for nearly a quarter of the hospital's grade.

Hospitals that do not complete the voluntary Leapfrog Hospital Survey also receive a Leapfrog Hospital Safety Grade, calculated entirely using publicly available measures.

The researchers ran simulations to see how these nonparticipating hospitals' grades might have changed if they had self-reported low, average or perfect Safe Practices measures while their infection rates remained the same.

Even if their performance on infections is good, such hospitals would probably end up with lower Leapfrog Hospital Safety Grades if their performance on the self-reported Safe Practices was anything less than perfect, researchers found.

In other words, hospitals that self-report less-than-perfect Safe Practices scores are probably better off not completing the Leapfrog Hospital Survey.

Shawna Smith, Ph.D., the study's lead author, explains, "Hospitals that have less-than-perfect performance on Leapfrog's list of protocols get a big hit to their grade, but those with most or all protocols in place are rewarded very little. Our simulations show that some hospitals would have had better grades when they didn't report than when they reported imperfect compliance."

Smith is a U-M Internal Medicine statistician and Institute for Social Research postdoctoral fellow.

This disincentive also creates a problem for moving hospitals forward on Safe Practices. "One of the principles of promoting Safe Practices is to ensure that hospitals are comfortable reporting that they still have work to do," says Meddings, a member of U-M's Institute for Healthcare Policy and Innovation who also holds a position at the VA Center for Clinical Management Research.

A path forward for a pioneering effort

The authors note that Leapfrog has encouraged public reporting on patient safety by hospitals for 16 years — dating back to when the public had little or no access to information about how safe their hospitals were. The paper gives a timeline of public reporting by Leapfrog and others.

Now, more than half of hospitals that complete the Leapfrog survey report perfect performance on all Safe Practices measures. The researchers say it's time to put more weight on publicly reported safety measures from objective sources rather than self-reporting of protocols.

While the voluntary Leapfrog hospital survey is free, hospitals are encouraged to use their grades to promote their hospitals to patients and potential patients — but they must pay licensing fees to use the grades as a marketing tool.

"The Leapfrog approach was really cutting-edge when it first came out and started an important movement," says Meddings. "But Leapfrog relies in part on process measures which can top out as a new process becomes widely adopted. It's time to have a new discussion about how to measure new metrics and use them in innovative ways to evaluate hospitals."

Adds Smith, "Leapfrog has a number of well-regarded experts involved, and their use of the Safe Practices measures does add something different to the universe of metrics out there. But they need to think how they are used in calculating the grade, because the current methodology penalizes hospitals for reporting scores that are anything less than perfect. If we want to encourage hospitals to continually improve safety, we need to incentivize acknowledging they have work to do."

Disclosure: The University of Michigan's academic medical center, Michigan Medicine, participates in the Leapfrog survey.

The research was funded by the Agency for Healthcare Research and Quality (HS19767, HS018334). Co-authors are Heidi A. Reichert, M.A., and Jessica M. Ameling, M.P.H.