Augmented Medicine Lung Diagnosis
The application enables a learning environment that realistically simulates the process of physical examination of patients presenting with shortness of breath. It hereby allows medical students and doctors to train themselves in diagnosing patients with lung diseases across a wide range of scenarios.
The HoloLens is used to place a hologram of an anatomic model of the human torso on top of the body of a real person. By using a real stethoscope that is being positionally tracked by the HoloLens, students can interact with the virtual torso. As they touch the hologram with the stethoscope, an audio sample of a lung sound that was recorded from a real patient is played back. The type of audio recording that is played back is determined by the position at which the stethoscope touches the hologram. By this, students learn to distinguish sounds of various lung diseases and learn how the sounds they hear relate to the position at which they place the stethoscope on the body. Since the application is networked, it can be used in a social learning setting where groups of students collaborate under the supervision of a teacher.
Next to developing the app, we created a gamified scenario for integrating the app into the medical curriculum. In this scenario, students meet an actor in the role of lung patient. The virtual torso is projected on the actor's body. By interviewing the actor and requesting additional information sources, such as radiology images and a patient file, students have to come up with the correct diagnosis and formulate a plan for treatment of the patient. By this, students can practice the full process of diagnosing lung diseases in a simulated environment that integrates many of the social, embodied and multimodal components of a real-world setting.
Marlies Reinders, Arianne Pieterse, Beerend Hierck, Franka Luk, Jelger Kroese
App and UI design:
3D model design:
Franka Luk, Arianne Pieterse, Beerend Hierck, Jelger Kroese
Franka Luk, Leontine van Melle