Marshall Allen was surprised, to say the least, when he was named a "Top Doctor" in America—mostly because he's not a doctor at all.

What's the actual methodology behind quality award programs?

Writing for ProPublica, Allen, a health care journalist, says the award made him grow "curious" about how recipients of these and similar awards are chosen. What he uncovered was a sometimes-shady world of awards that, in some cases, are little better than scams.

5 ways to strengthen your physician leadership development program

Marshall Allen, 'Top Doctor'

Last year, Allen writes, he received a call from Top Doctor Awards saying he'd been chosen to receive a "Top Doctor" Award. When Allen asked how he had been selected, he was informed that he'd been nominated by his peers and reviewed by his patients.

There's one problem: Allen isn't a doctor. "I don't have a medical degree and I'm not a physician," he writes. "But I am an investigative journalist who specializes in health care."

Allen informed the contact from Top Doctor of his professional background, and he asked if that prevented him from receiving the award. Apparently, it wasn't a problem: Allen could still receive the award, and it even came with a customized plaque.

"There's a nominal fee for the recognition," the person on the phone told him. "It's a reduced rate. Just $289."

That stopped Allen in his tracks. But the Top Doctor contact pressed on: "The plaque commemorates your achievements and more importantly communicates the achievements to your patients. It's a great achievement. I would hate for you to miss it. I can get it to you right now for $99."

Allen accepted the offer—although, as he archly writes, "Obviously, the Top Doctor Awards company has questionable standards."

How companies determine who wins their awards

This experience led Allen to investigate how companies like Top Doctor Awards determine who's worthy to receive these honors. "The more I looked, the more companies I found heaping praise on doctors and then charging them to market the honor," he writes, adding, "There's Castle Connolly Top Doctors, Super Doctors, The Best Docs, and many more."

Spokespeople for several of the awards defended their selection process. John Connolly, co-founder of the company that releases the Castle Connolly Top Doctor awards, said the company depends on nominations from other physicians to determine its "top doctors." Connolly also said the company has a research team that checks the license, board certification, education, and discipline history of each nominee.

Your data-driven road map for physician engagement

Connolly added that he thinks it would be "very difficult" for anyone to game the nominations. However, he added, "We don't claim [the doctors] are the best. We say they are 'among the best' and ones we have screened carefully."

Super Doctors, another awards company, offered a similar disclaimer, Allen writes. "No representation is made that the quality of the medical services provided by the physicians listed in this web site will be greater than that of other licensed physicians," the company said.

As for Top Doctor Awards, Donna Martin, who works in client services for the company, said winners of the "Top Doctor" awards may be nominated by their peers because of their achievements. She claimed, however, there's also a "full interview process," which includes questions about the nominee's other awards, their education, and their leadership. But Allen notes that he did not undergo an interview for his award.

Allen later called Martin and asked how good the company's methods could be if he received a "Top Doctor" award. She said she would call him back—but hasn't yet, Allen writes.

What health experts think of the awards

Michael Carome, the director of Public Citizen, a health research and advocacy organization, didn't mince words about his feelings on these types of awards. "This is a scam," he said. "Any competent qualified doctor doesn't need one of these awards unless they want to stroke their ego. These are meaningless, worthless awards." Carome added that he believes it's unethical to hand out or accept these awards.

John Santa, who spent years measuring health care quality for Consumer Reports magazine, said that any award process that relies on nominations will be rigged by doctors with financial ties to each other. "If you look at these methodologies, they are rife with economic and relationship biases," he said.

Santa added that he believes these awards are an insult to patients, who deserve better information about their providers. Many of these awards companies do little more than verify a doctor's credentials with a state board, Allen writes. "I'm sorry, being in good standing with a state licensing board is a very low bar," Santa said. "Being board certified is a very low bar."

Allen also called up some other "Top Doctor" award winners. While most didn't return his calls, one did: Lewis Maharam who practices sports medicine in New York City.

Maharam told Allen he wasn't surprised he received a "Top Doctor" award because "he has a wall filled" with similar awards.

But when Allen informed Maharam that he, a non-physician, had also received a "Top Doctor" award, he changed his tone. "That's pretty strong evidence that it's not legitimate," Maharam said. "It might have been that my assistant just sent in a check." Maharam explained that doctors are often so busy it can be easy to take advantage of them. 

Maharam added that, in light of Allen's story, he may not pay the fee to renew his "Top Doctor" award this year (Allen, ProPublica, 2/28).

What's the actual methodology behind quality award programs?

Download our one page sheets for summaries on the methodology and metric categories used in seven hospital quality rating programs:

• CMS Overall Star Ratings
• CMS HCAHPS Patient Experience Star Rating
• Healthgrades Patient Safety Ratings and Excellence Awards
• Healthgrades Outstanding Patient Experience Awards
• Leapfrog Group Hospital Safety Grade
• Watson Health 100 Top Hospitals
• US News and World Report Best Hospital Rankings

Get all the Cheat Sheets