Educational Research Column: Using Surveys in Medical Education Research

Educational Research Column: Using Surveys in Medical Education Research

Using Surveys in Medical Education Research

Surveys are a common means of acquiring data in medical education research. They can be used to perform needs assessments for curriculum design, evaluate learner and instructor perceptions, or gather information about the attitudes and preferences of individuals in the educational environment.

Surveys are best used to obtain subjective data, such as opinions or attitudes, as these cannot be observed directly. Directly observable data may be gained by various means, such as direct measurement (e.g. using an exam to test content retention), database review (e.g. analyzing student academic records), or third-party observation (e.g. outsourcing analysis of videotaped procedural performance). In contrast, surveys ask respondents to report or recall information, and as such their findings risk being subject to various cognitive biases such as recall bias. For this reason, data should only be obtained by a survey if they are otherwise unavailable due to the impracticality or impossibility of more direct assessment.

When designed and applied properly, surveys can achieve a scientific rigor comparable to psychometric tools. If relevant to the researcher’s questions, previously validated survey tools should be used, as these are more likely to yield high-quality responses and allow for comparison against pre-existing data. Many questions in medical education research may require the construction of new surveys, however, as pre-existing survey questions may not fully capture the information relevant to the study at hand.

Artino et al. describe seven steps in developing a survey tool to maximize its scientific rigor: (1) conduct a literature review to look for pre-existing survey tools or items, and to align the new survey with existing literature; (2) conduct interviews or focus groups to help understand how the population of interest conceptualizes the relevant questions; (3) synthesize the first two steps to ensure that the ideas under investigation a) use language that the population of interest can understand, and b) that these ideas make theoretical sense based on the literature; (4) develop survey items; (5) conduct expert review of these items to assess for content validity; (6) vet these items with members of the respondent population to ensure understanding in the manner that the investigators intend; and (7) run a pilot survey to evaluate response variance and conduct a formal analysis of the content validity of each survey item. While the performance of each of these steps may be too cumbersome for the construction of a given survey, incorporating as many as possible will increase the quality and reproducibility of survey responses and will make the data more reliable for comparison against future studies.

Furthermore, careful design facilitates the reporting of many important survey considerations. A robust evaluation of the study population will enable clearer description of sampling criteria, which will add to a survey’s reproducibility and indicate the generalizability of its results. Content review and item vetting not only improve the quality of data collected by individual questions, but also help identify the best means of response measurement. For example, an opinion question might utilize a Likert scale or free responses, and these two types of data demand very different analyses. Identifying which response type is best suited to gathering the data of interest is crucial for both data acquisition and reporting.

Finally, carefully crafted survey items avoid ambiguity or confusing language. This helps reduce erroneous responses, and thus makes inaccurate conclusions less likely. Clear wording can also improve the response rate, which is a crucial piece of information for any survey study to report. As in all study types, achieving appropriate power helps reduce the risk of type I and type II errors, and in survey methodology power is reflected by the response rate as well as the sample size.

Surveys can be excellent tools for elucidating subjective or otherwise non-observable data. With careful design and appropriate application, they can attain a high degree of scientific rigor.

Andrew Ketterer, MD
Medical Education Fellow
Beth Israel Deaconess Medical Center/Harvard Medical School

References

• Artino AR, La Rochelle JS, Dezee KJ, et al. Developing questionnaires for educational research. AMEE Guide No. 87. 2014(36):463-74.
• Phillips AW. Proper applications for surveys as a study methodology. Brief Educational Advances. December 5, 2016. 18(1).