Do Words Matter? A Linguistic Analysis of Letters of Recommendation for Residency at a Large Academic Plastic Surgery Program
Helen Liu, BS; Daniel Ranti, BS; J. Roscoe Wasserburg BA; John W Rutland BA; Abigail Katz BA; Yasmina Zoghbi MD; Farah Sayegh MD; Peter J Taub MD, MS
Mount Sinai Hospital, New York, NY, USA
Background: Cognizant that Step 1 of the USMLE will adopt a pass/fail format, the letter of recommendation (LOR) will become an increasingly important component of the Match, and whether it should be standardized remains highly debated. The American Council of Academic Plastic Surgeons (ACAPS) developed a standardized LOR that allows letter writers to rank applicants. Previous studies have investigated components of a successful application but have omitted the LOR. This analysis seeks to understand the utility of the free-text LOR as it may relate to the request for an interview.
Methods: 489 LORs from 132 applicants were gathered at a single integrated plastic surgery residency program from the 2019-2020 application cycle. Data was gathered from the complete ERAS application form, including subjective standardized ranking scores and details about the letter writer. Group comparisons were performed between interviewed candidates and non-interviewed candidates. Linguistic and regression models were used to estimate the LORís utility in contributing to the receipt of an interview. Feature importance, utility of certain words in predicting a candidate's interview status, was determined using model weights.
Results: The following variables were statistically significant: letter writer location, duration of relationship between the letter writer and applicant, NIH funding, H-index of the writer, research and teaching percentile of the student, and academic skills percentile of the student. Of the 10 standardized LOR ranking variables, only research and academic skills were statistically significant. Using text only, the BOW model achieved higher performance than the regression model (AUROC 0.76 vs 0.71). The most highly weighted feature in the regression was letter probability.
Conclusion: The results suggest that the unstructured LOR has more predictive ability than standardized rankings. The analysis suggests that both the letter writer and content of the unstructured letter substantially impact chances of receiving an interview. In addition, text analysis and quantitative comparisons show that research and leadership highly contribute to the chances of receiving an interview. In summary, the present analysis underscores the importance of the unstructured letter when compared with standardized elements of the recommendation, emphasizing that standardizing the LOR may offer little differentiation between candidates.
Back to 2022 Abstracts