| We are pleased to announce the addition of a new section in the Journal focusing on performance assessment. The use of performance assessment in the training and evaluation of health professionals has grown rapidly over the past decade. We believe this section will provide a useful outlet for disseminating research and innovations in this expanding field. For the purposes of this section, performance assessment is defined as the evaluation of the ability of a healthcare provider or a healthcare team to demonstrate the skills necessary to provide health care in a real-world setting requiring a variety of knowledge, skills, attitudes and professional behaviors. The new section will include manuscripts addressing the use of performance assessment in healthcare education and evaluation including research, descriptions of innovations, software/ hardware evaluations, and commentary on the use of performance assessment. We are very pleased to announce that Brian Mavis, PhD, has agreed to serve as Editor of MEO’s Performance Assessment Section. Dr. Mavis has published extensively in the area of performance assessment and is a long-time member of the Editorial Board. He is Director of the Office of Medical Education Research and Development in the College of Human Medicine at Michigan State University. Dr. Mavis serves on the advisory committees of the Graduation Questionnaire and the Medical Education Research Certificate program for the Association of American Medical Colleges (AAMC). Dr. Mavis is a past member of the AAMC Research in Medical Education (RIME) program planning committee and the National Board of Medical Examiners (NBME) Medical School Advisory Board. He chairs the Executive Committee for the AAMC Central Group on Educational Affairs. We also welcome several new members of the MEO Editorial Board to expand our expertise in performance assessment along with the broader field of medical education. These include Clarence Kreiter, PhD (College of Medicine, University of Iowa), Cathy Lazarus, MD, (Chicago Medical School at Rosalind Franklin University of Medicine and Science), Emil Petrusa, PhD, (Vanderbilt University School of Medicine), Gominda Ponnamperuma, MBBS, PhD (Centre for Medical Education, University of Dundee), and Lynn Yeoman, PhD (Baylor College of Medicine). We would also like to express our gratitude to our long-time editorial board members who have devoted their time and expertise in serving the Journal alongside the dozens of medical educators who provide constructive feedback on the more than 80 manuscript submissions MEO receives each year. We are pleased to be able to launch the new Performance Assessment Section with the three excellent articles listed below that exemplify the types of manuscripts we hope to publish in the future. We look forward to receiving your submissions for this new section. David Solomon, PhD and Ann Frye, PhD Editors, Medical Education Online A report on the piloting of a novel computer-based medical case simulation for teaching and formative assessment of diagnostic laboratory testing Clarence D. Kreiter, Thomas Haugen, Timothy Leaven, Christopher Goerdt, Nancy Rosenthal, William C. McGaghie and Fred Dee Abstract Objectives: Insufficient attention has been given to how information from computer-based clinical case simulations is presented, collected, and scored. Research is needed on how best to design such simulations to acquire valid performance assessment data that can act as useful feedback for educational applications. This report describes a study of a new simulation format with design features aimed at improving both its formative assessment feedback and educational function. Methods: Case simulation software (LabCAPS) was developed to target a highly focused and well-defined measurement goal with a response format that allowed objective scoring. Data from an eight-case computerbased performance assessment administered in a pilot study to 13 second-year medical students was analyzed using classical test theory and generalizability analysis. In addition, a similar analysis was conducted on an administration in a less controlled setting, but to a much large sample (n=143), within a clinical course that utilized two random case subsets from a library of 18 cases. Results: Classical test theory case-level item analysis of the pilot assessment yielded an average case discrimination of 0.37, and all eight cases were positively discriminating (range=0.11-0.56). Classical test theory coefficient alpha and the decision study showed the eight-case performance assessment to have an observed reliability of σ=G=0.70. The decision study further demonstrated that a G=0.80 could be attained with approximately 3 h and 15 min of testing. The less-controlled educational application within a large medical class produced a somewhat lower reliability for eight cases (G=0.53). Students gave high ratings to the logic of the simulation interface, its educational value, and to the fidelity of the tasks. Conclusions: LabCAPS software shows the potential to provide formative assessment of medical students’ skill at diagnostic test ordering and to provide valid feedback to learners. The perceived fidelity of the performance tasks and the statistical reliability findings support the validity of using the automated scores for formative assessment and learning. LabCAPS cases appear well designed for use as a scored assignment, for stimulating discussions in small group educational settings, for self-assessment, and for independent learning. Extension of the more highly controlled pilot assessment study with a larger sample will be needed to confirm its reliability in other assessment applications. Standardized patient outcomes trial (SPOT) in neurology Joseph E. Safdieh, Andrew.L. Lin, Juliet Aizer, Peter.M. Marzuk, Bernice Grafstein, Carol Storey-Johnson and Yoon Kang Abstract Background: The neurologic examination is a challenging component of the physical examination for medical students. In response, primarily based on expert consensus, medical schools have supplemented their curricula with standardized patient (SP) sessions that are focused on the neurologic examination. Hypothesis-driven quantitative data are needed to justify the further use of this resource-intensive educational modality, specifically regarding whether using SPs to teach the neurological examination effects a long-term benefit on the application of neurological examination skills. Methods: This study is a cross-sectional analysis of prospectively collected data from medical students atWeill Cornell Medical College. The control group (n=129) received the standard curriculum. The intervention group (n=58) received the standard curriculum and an additional SP session focused on the neurologic examination during the second year of medical school. Student performance on the neurologic examination was assessed in the control and intervention groups via an OSCE administered during the fourth year of medical school. A Neurologic Physical Exam (NPE) score of 0.0 to 6.0 was calculated for each student based on a neurologic examination checklist completed by the SPs during the OSCE. Composite NPE scores in the control and intervention groups were compared with the unpaired t-test. Results: In the fourth year OSCE, composite NPE scores in the intervention group (3.5±1.1) were statistically significantly greater than those in the control group (2.2±1.1) (p<0.0001). Conclusions: SP sessions are an effective tool for teaching the neurologic examination. We determined that a single, structured SP session conducted as an adjunct to our traditional lectures and small groups is associated with a statistically significant improvement in student performance measured 2 years after the session. Better data for teachers, better data for learners, better patient care: college-wide assessment at Michigan State University’s College of Human Medicine Aron C. Sousa, Dianne P. Wagner, Rebecca C. Henry and Brian E. Mavis Abstract When our school organized the curriculum around a core set of medical student competencies in 2004, it was clear that more numerous and more varied student assessments were needed. To oversee a systematic approach to the assessment of medical student competencies, the Office of College-wide Assessment was established, led by the Associate Dean of College-wide Assessment. The mission of the Office is to ‘facilitate the development of a seamless assessment system that drives a nimble, competency-based curriculum across the spectrum of our educational enterprise.’ The Associate Dean coordinates educational initiatives, developing partnerships to solve common problems, and enhancing synergy within the College. The Office also works to establish data collection and feedback loops to guide rational intervention and continuous curricular improvement. Aside from feedback, implementing a systems approach to assessment provides a means for identifying performance gaps, promotes continuity from undergraduate medical education to practice, and offers a rationale for some assessments to be located outside of courses and clerkships. Assessment system design, data analysis, and feedback require leadership, a cooperative faculty team with medical education expertise, and institutional support. The guiding principle is ‘Better Data for Teachers, Better Data for Learners, Better Patient Care.’ Better data empowers faculty to become change agents, learners to create evidence-based improvement plans and increases accountability to our most important stakeholders, our patients. |
Medical Education Online eISSN 1087-2981
This journal is published under the terms of the Creative Commons Attribution-Noncommercial 3.0 Unported License. Responsible editor: David J Solomon.