MGB’s quality and patient experience leaders recognized that spending money on infrastructure for an ePRO program would be futile if the organization didn’t first invest in a culture that would accept the program. Consequently, MGB intentionally invested in the following three factors while developing their program.

Factor 1: Buy-in

Leaders went department by department to identify physician champions for the PROMs program. Physician champions helped to create buy-in among their peers and quicken the pace of the program’s rollout. For example, departments with strong physician champions stood up the program within a few months, whereas departments without champions took a few years to implement the program. As part of this effort, leaders educated physicians about how ePRO data collection improves patient outcomes and how the data can support physicians during the patient encounter. Leaders also emphasized to physicians that the program would not significantly impact physician workflows.

Sample talking points for building buy-in among physicians:

• “ePROs can provide additional context about the patient before walking into the exam room.”
• “ePRO data can save you time during the patient encounter since ePROs identify the patient’s key problems quicker and earlier.”
• “ePRO data provides timely insights about patients at the point of care and equips you with data to answer and support patient questions.”
• “ePROs help compare where the patient is today versus where you might expect them to be.”

Factor 2: Staffing

Simultaneously, leaders created a dedicated team of individuals to roll out and support the program. MGB invested in building a PROMs department, which includes program managers, content specialists, EHR analysts, and clinic support staff. Today, MGB has almost 10 FTEs to support their enterprise-wide PROMs initiative that spans 90 specialties and 213 clinics. This central team is responsible for building patient questionnaires, managing the data warehouse, and training and supporting providers and staff.

Advice from MGB

Programs need to first invest in people and technology to then build the infrastructure. You need to have dedicated FTEs and physician champions who can get providers on board and continually show providers how to use the data patients are providing.

Dr. Nadine Jackson McCleary

Mass General Brigham

Factor 3: Technology

MGB invested in the necessary infrastructure (technology, internet, etc.) for their program. MGB worked with their EMR vendor to operationalize workflows, coordinated with IT and tablet manufacturers, and upgraded their WIFI in many parts of the institution to support in-office tablet distribution.

With these three components in place (organizational buy-in, dedicated staff, and the required technology), MGB then rolled out the program across each specialty that had chair and practice manager signoff as well as physician engagement.

To be successful, an ePRO program must balance provider and patient ease of use and clinical utility. Here are some of MGB’s guiding principles for patient questionnaires to achieve that outcome.

Just because you can ask doesn’t mean you should

MGB’s questionnaires are validated instruments, psychometrically tested, andassociated with improved patient outcomes. Consequently, every question mustbe clinically meaningful and tied to an improvement in outcomes. This meansthat just because a question is interesting doesn’t mean it should be asked.

Ask questions when they clinically matter

MGB first assesses when a patient would expect to see a clinical improvement ordeterioration, and then sends questionnaires at or before that point. The ePROdata collected then allows for benchmarking of a patient’s subsequent statusagainst a patient’s initial status. Providers can then gauge improvement over timefrom the point of intervention or treatment.

Respect patients’ time

Health is only one facet of patients’ lives, so MGB intentionally accounts for patient burden and limits the number of questions patients are being asked. MGB questionnaires do not exceed 30 questions.

With these patient questionnaire principles in mind, MGB implemented the belowworkflow to execute their PROMs program. Patients fill out simple, well-timed questionnaires via the patient portal at home or via tablets in the patient waiting room before their appointment. Providers can then access patient responses during the visit via the EMR to inform the patient encounter.

Traditionally, ePRO programs collect an individual patient’s data to inform that patient’s care. MGB takes this model a step further and provides physicians with both individual and aggregated ePRO data, which provides context to support clinician decision-making and conversations with patients. Physicians can harness ePRO data to enrich the patient encounter in the following ways:

Contextualize patient symptoms: When a patient reports new symptoms via the questionnaire, their physician can use the data to determine whether the patient’s symptoms are in line with those experienced by other patients during that point in treatment. The provider can then decide to adjust the patient's care plan or share the aggregated ePRO data view with the patient to reassure them that they are not alone in experiencing those symptoms.

Improve patient education: Patient education is critical to patient experience, yet providers often feel ill-equipped to engage in these conversations. Aggregated ePRO data gives physicians the benchmarks to set patient expectations for treatment and recovery. Before a treatment starts, providers can align a patient with what they should expect 30- or 90-days after treatment based on the outcomes data of other patients.

Train physicians on how to use ePROs for patient care

Physicians aren’t traditionally trained in medical school on how to interpret ePRO data, so MGB offers group trainings to make ePRO data as easy for providers to leverage as possible.

When new questionnaires are launched, providers receive trainings through forums and best practice publications. Key elements of the training include understanding the value of ePROs, the workflows for patients and providers, and how to read and consider both individual and aggregated ePROs in the EMR.

Within the EMR, MGB provides easy-to-understand and easy-to-use visualizations to make the ePRO data meaningful to providers. The intention is that providers shouldn’t have to be data experts to interpret ePROs.

Advice from MGB

When a patient shares an experience, physicians often believe they have the best knowledge to interpret it. We are learning this is misguided thinking. What we see in the clinic is different from what patients are experiencing when they are not with us. PROs help us capture what patients are really experiencing—the real-world data.

Dr. Nadine Jackson McCleary

Mass General Brigham

MGB aims to ensure that the ePRO data influencing care delivery isn’t coming from a homogenous patient sample but a group representative of the diverse patient populations they serve. This required MGB to identify inequities in access that exist in both the technology platform and operations of the program.

Below are three examples of the changes MGB made to ensure their PROMs program was inclusive.

Build for patients with the greatest need

Many digital health tools are designed for patients who have access to health care and few barriers to digital technologies. However, MGB leadership recognized that in order to improve outcomes for all patients, they would need to build for their highest-need patients, especially those without access to specific technologies, and/or who lack digital or health literacy. The ePRO team now consistently reflects on the question of “who is the system built for?” and challenges themselves to build for patients who might be outliers or have the greatest need.

Advice from MGB

If ePRO data is coming from a homogenous population, you can’t use it to make predictions about everyone.

Dr. Nadine Jackson McCleary

Mass General Brigham

Think beyond digital inequities

Often when organizations implement PROMs programs, they focus on addressing digital inequities (e.g., access to smart phones, tablets, internet connectivity). However, it is equally as important to account for the languages and literacy levels of the patients being served. For example, MGB’s patient questionnaires were initially offered only in English and were inaccessible to patients who were visually or cognitively impaired.

In 2020, MGB embarked on a system-wide initiative (United Against Racism) to close disparities and increase health care equity. As part of this effort, MGB’s PROMs program has now translated and built over 500 questionnaires in seven different languages (Spanish, Portuguese, Traditional Chinese, Haitian, Creole, Arabic, and Russian). Additionally, dedicated coordinators are now available to help patients fill out their questionnaires as needed.

Identify and address bias in operations

MGB discovered that bias can exist not just in ePRO platforms and questionnaires but also in survey distribution channels. For example, MGB found that their front desk staff were less likely to offer ePRO tablets to patients of color compared to white patients. Program leaders then met with staff to discuss the problem. By simply pointing out the disparity and providing education, MGB increased tablet distribution from 30% to 86% in two months.

Standing up a PROMs program is an iterative process. To continuously improve the program, PROMs program staff conduct quality assurance checks to ensure program accessibility, usability, and ease. The team tracks a multitude of metrics to evaluate whether patients are appropriately engaging with the program, providers are using the ePRO data to inform the patient encounter, and the organization is using the data to better target quality improvement initiatives.

The quality assurance checks help improve the overall effectiveness of the PROMs program, but they also serve to guide how MGB defines what success looks like—and how the definition of success might evolve. Below is MGB’s current vision for what a successful PROMs program should accomplish.