Formative Research Summary for Lift

We completed Formative Research to inform the development of Lift simulations, cases, and online information from 2018 to 2022. This is a summary of the majority of testing, which was completed with medical students.

Phase I Usability and Preference Testing:

We sought medical student design preferences and qualitative feedback regarding simulation engagement, interactions, pacing, appearance, and interest. Testing was iterative in order to successively refine the framework and prototype. Feedback guided us towards a modern appearing clinic, with realistic details from clinics such as the need for more characters walking around, adding clinic background noises to the audio, using fairly high-quality graphics, and being able to teleport across space rather than having to walk between areas of the clinic.
Feedback from each round of testing led to alterations in, first, the sim design document and then the prototype, using an agile process; revisions were made after testing yielded consistent results. For example, students slightly preferred freedom to navigate a 3D environment over linear navigation of a 2D environment. However, many preferred to play a 2D version online than to download a 3D simulation. In response, we constructed the simulation experience in both formats. We tested the prototype at mid-development with 8 medical students, mostly in their 4th year with the following results:

Usability Test of Phase I Prototype

11/11-12/1/18 (n=8 medical students; 4th year (n=6), 1st year (n=1), Unknown year (n=1).

Needs Analysis Questions (5-point Likert Agreement Scale)

• 87.5% agreed that medical school is stressful. (Likert mean: 3.9)
• 87.5% were interested in learning more about stress, coping, and wellness as they relate to medical school/being a physician. (Likert mean: 4.4)

Usability/Alpha Testing Questions (5-point Likert Agreement Scale)

100% agreed that the topics were relevant to them (mean: 4.2)
75% agreed that they were interested in the storyline (mean: 3.8)
75% agreed that the feedback they received about choices was useful. (mean: 3.5)
62% agreed that they enjoyed playing the simulation (mean: 3.5)
62% agreed they would recommend the simulation experience to others (mean: 3.4)
62% agreed they want to use a VR version of the simulation (mean: 3.4)
Half of the students (50%) agreed the simulation could be a useful learning experience, helped them learn how to avoid burnout, could help them learn to cope with stress, or were interested in playing further scenarios. This information plus details shared by students in open-ended feedback, guided development to improve on these issues during Phase II.
Few students (38%) preferred a phone/mobile version (mean 3) or two read about the topic on a website (mean 2.8)

Focus Group Needs Analysis Phase II (n=8)

In Phase II, we conducted needs analysis online focus groups with two groups of 4 medical students. They represented clinical and preclinical stages. The topic was: “Challenges that could lead to symptoms of burnout in medical students, advice and skills that could help prevent those symptoms, and perceptions of whether anticipatory guidance would be effective.” The first group gave open-ended feedback which generated the topics covered in a more structured discussion of the second group.

Results for Structured Discussion (n=4):

The four students were unanimous in their opinion that burnout, with the primary symptoms of being overwhelmed and exhausted mentally, emotionally, physically, and spiritually and outcomes of low self-esteem, discouragement, and unhappiness with daily lives are still a big problem for the majority of medical students. Three participants experienced all of these symptoms themselves, and one experienced only physical exhaustion. “Depression” was not a symptom for these participants.

The students felt that there was not a single, major stressor. Instead, they said the problem was that there were many mild to moderate stressors that add up to produce major stress. The stressors they identified were:

An overwhelming volume of work. In pre-clinical years, it is how much there is to learn in class work and simultaneously having to study for Step 1 exams. The classwork does not cover all of what students need to know for the exams so it requires additional learning.
A culture of competitiveness. Their peers are very competitive, but students believe the medical education system and all of medicine supports this.
Pressure to do what is needed to be able to get into the residency they want. There are more things to do (leadership roles, research, elective activities) than they have time, and there is uncertainty about which to choose.
A rigid, not open-minded, not supportive-enough-academic system that does not respond well to different learning styles, ideas from students for solutions, and special circumstances in medical education as a whole. This was contrasted with undergraduate education which was more flexible.
Expectations of parents and others in their lives.
Lack of self-care, a common response to trying to handle the first several stressors, becomes a stressor itself, as the effects of lack of sleep, poor eating, little exercise, etc. take their toll.
Work-life balance importance is something they know, y, but they worry that achieving it will ruin their academic performance. They find it hard to trust that balance might even enhance their academic performance.
Even after achieving work-life balance, maintaining self-care, and the ability to judge self-worth based on their value systems rather than the culture of competitiveness, students still were burned out due to the rigidity of the academic system and demanding workload.

Clinical training stressors were:

Time management/work-life balance. Students spend 8 or more hours in the clinic and then come home and have to study to keep up with classes. After doing this for a while, students start to burn out from not having a break.
Administrative demands on top of having to deal with patients. These include learning complex hospital protocols and the EHR for each rotations site, dealing with insurance, and obtaining patient records from other institutions.
Some of the experiences working with patients, such as patients who are not happy about having a medical student work with them, patients who do not cooperate in talking about their history, and patients who tell them one history and then tell the attending another. Meeting all of a patient’s needs in the time allotted and wishing to go above and beyond to help a patient but not having sufficient time was another stressor related to patient care.
Unpleasant interactions with attending physicians when students make a misstep or do not know something. Some attendings behave poorly in interactions with medical students because of their own stressors, assume incorrectly that the student has a bad attitude, does not care, or is unwilling to do hard work when instead it is typically that students do not know or understand something or there are circumstances the attending does not know. The residency application system adds further stress to these interactions because of the desire to obtain a good recommendation from the attending.
The pressure to already know more medicine than students can possibly know. They feel a need to appear as if they know more than they do. They spend much of their time trying to learn it all.
An under-represented minority student added: Being an underrepresented minority in medicine is a cause of extra stress as there are microaggressions and preconceived notions that attendings, residents, hospital staff, and patients have that you must overcome; it can be difficult to juggle the stresses of being a medical student plus imposter syndrome plus struggling with the idea of having to be the model minority.

Regarding the design for the simulation, students recommended:

All agreed that certain anticipatory guidance would be helpful.
They felt the timing of guidance was important in that it needs to come close to the stressful event or even a little after it has started.
They liked the idea of a simulation going through a typical day for a medical student so that the many small stressors could be simulated.
They wanted a simulation that allows for experiencing a successful response to each challenge.
They felt the simulation should also depict proactive actions (not just responding reactively to challenges), such as seeking out a mentor or peer support, going for counseling, getting exercise, taking time for enjoyment, etc.
When asked if they would like a mentor NPC in the game, they seemed to like that idea.
They had plenty of information about what medical school would be like ahead of time, but they did not “get it” until they were in it. They worried that a too realistic simulation might discourage some potential doctors, since they were still all happy that they were in medicine.
They liked our idea of peer support training and liked that it would include skills they could also use with patients.

Anticipatory guidance the medical students wanted to share:

Consensus guidance:
1. Develop good study habits. This involves discovering your learning style and following tips for that style (n=2), doing a little bit every day and not putting it off, and then trying to cram all at once (n=1). Use small segments of time throughout the day to fit in short study sprints. Be careful not to peak too early in your Step 1 studies (n=1)
2. Don’t try to do every possible thing to enhance your residency application, such as joining special interest groups, taking a leadership position, or doing research. Seek advice from a mentor in your target specialty, such as a resident, about which ones to focus on.
3. Seek help early for emotional problems from friends and formal programs available in the school such as counseling. Seek help from available supports, such as peer support groups, buddy systems or mentoring systems, and school or private counseling. Medical students may not be used to needing help because many, until this point, had their successes come easily. `
4. Make peace with the competitive environment and your own internal competitive drive vs. living according to your values (e.g., not sacrificing relationship or family responsibilities) and taking care of yourself. All agreed that some of the pressure to compete was coming from inside themselves. They could use help with that and see it is not really needed. One said they need reminders to operate from an internal locus of control rather than internalizing the competitive culture. They want to know how to avoid getting caught up in it to the extent that it is not beneficial.
5. Seek a work-life balance. Don’t try to study endlessly. At some point, there are diminishing returns. You can take some time for yourself and still do okay or good enough. However, although everyone knew about this importance, they agreed that actually achieving it is more challenging. All of them came to the conclusion you just have to take time for yourself no matter what and trust that it will not ruin your academic performance and may even enhance it.
Guidance from individual participants:
1. Starting 4th-year student – Start building a routine now so that medical school becomes a new part of your life and does not take over it.
2. Starting 4th-year student – Be balanced – Make time to do the things you love to do. Give an honest effort – once you have done your best, accept the results. Keep a growth mindset – always look for areas where you can improve, which is not a sign of weakness but of great strength.
3. Starting 2nd-year student – Managing your physical, emotional, mental, and spiritual well-being is extremely important to your success in medical school. My favorite quote “You cannot pour into an empty cup” says it all, if you do not take care of yourself you cannot help others.
4. Starting 2nd-year student – Whenever you do anything add it to your CV right then and there so that when it comes time to apply to residency, you have it completed.

Later (Phase II) Usability Study

Subsequent iterative usability tests with a total of 13 medical students representing each year of school, helped us refine the simulations in terms of length of each interaction with a simulated peer student, length of each passage and mentor feedback, clarity, writing level, clarity of instructions, achievement of our mission, quality of the experience, realism of the storyline. We responded to lower ratings by making corrections, clarifications, and improvements. Qualitative data in response to a follow-up open-ended request for feedback helped us address specific aspects of the storyline that were insufficiently realistic.

These tests also helped us test and make minor modifications in the presentation and flow of the assessments for the Summative study, which is presented separately.