Pursuing Excellence The Power of Selection Science to Provide Meaningful Data and Enhance Efficiency in Selecting Surgical Trainees
ANNALS OF SURGERY
Authors: Gardner, Aimee K.; Dunkin, Brian J.
Abstract
Introduction: As current screening methods for selecting surgical trainees are receiving increasing scrutiny, development of a more efficient and effective selection system is needed. We describe the process of creating an evidence-based selection system and examine its impact on screening efficiency, faculty perceptions, and improving representation of underrepresented minorities. Methods: The program partnered with an expert in organizational science to identify fellowship position requirements and associated competencies. Situational judgment tests, personality profiles, structured interviews, and technical skills assessments were used to measure these competencies. The situational judgment test and personality profiles were administered online and used to identify candidates to invite for on-site structured interviews and skills testing. A final rank list was created based on all data points and their respective importance. All faculty completed follow-up surveys regarding their perceptions of the process. Candidate demographic and experience data were pulled from the application website. Results: Fifty-five of 72 applicants met eligibility requirements and were invited to take the online assessment, with 50 (91%) completing it. Average time to complete was 42 +/- 12 minutes. Eighteen applicants (35%) were invited for on-site structured interviews and skills testing-a greater than 50% reduction in number of invites compared to prior years. Time estimates reveal that the process will result in a time savings of 68% for future iterations, compared to traditional methodologies. Fellowship faculty (N = 5) agreed on the value and efficiency of the process. Underrepresented minority candidates increased from an initial 70% to 92% being invited for an interview and ranked using the new screening tools. Discussion: Applying selection science to the process of choosing surgical trainees is feasible, efficient, and well-received by faculty for making selection decisions.
Investigating The Effect of Exposure-Control Strategies on Item Selection Methods in MCAT
JOURNAL OF MEASUREMENT AND EVALUATION IN EDUCATION AND PSYCHOLOGY-EPOD
Authors: Mao, Xiuzhen; Ozdemir, Burhanettin; Wang, Yating; Xin, Tao
Abstract
This study aims to investigate the effect of different item exposure controlling strategies on item selection methods in the context of multidimensional computerized adaptive testing (MCAT). Additionally, this study aims to examine to what extend the restrictive threshold (RT) and the restrictive progressive (RPG) exposure methods suppress the item exposure rates and increase the exposure rates of underexposed items without losing psychometric precision in MCAT. For this purpose, the performance of four item selection methods with and without exposure controls are evaluated and compared so as to determine how results differ when item exposure controlling strategies are applied with Monte-Carlo simulation method. The four item selection methods employed in this study are D-optimality, Kullback-Leibler information (KLP), the minimized error variance of linear combination score with equal weight (V1), the composite score with optimized weight (V2). On the other hand, the maximum priority index (MPI) method proposed for unidimensional CAT and two other item exposure control methods, that are RT and RPG methods proposed for cognitive diagnostic CAT, are adopted. The results show that: (1) KLP, D-optimality, and V1 performed better in recovering domain scores, and all outperformed V2 with respect to precision; (2) although V1 and V2 offer improved item bank usage rates, KLP, D-optimality, V1, and V2 produced an unbalanced distribution of item exposure rates; (3) all exposure control strategies improved the exposure uniformity greatly and with very little loss in psychometric precision; (4) RPG and MPI performed similarly in exposure control, and outperformed RT exposure control method.